We asked learners on CareerVillage how they felt about AI-generated advice. Here’s what we heard.

Across the entire internet, there has been a significant increase in AI-generated content, driven by the era of LLMs. Since the fall of 2023, our team has observed an increase in the number of responses on the CareerVillage platform that contains AI-generated or AI-assisted content (or ‘Human-in-the-loop content’, as it’s termed in our Guidelines). 

In tracking and reviewing AI use on our platform, we hope to advance our understanding of this new and ever-changing technology, and ultimately, inform the ongoing development of our AI Policy, as well as the development of Coach, the AI-powered career development platform we’re launching this summer.

To gain a deeper understanding of community perspectives on the use of AI-generated and AI-assisted content in response to student questions, we took the additional step of conducting an 8-question survey among two segments in April 2024. 

1. CareerVillage Learners: 1,384 learners who created accounts in March 2024 were invited to complete the survey. To incentivize engagement, the first ten users to complete the survey were sent a $10 gift card. Ultimately, we received 102 completed survey responses from this segment.
2. Early Adult Participants: Conducted among 60 English-fluent participants aged 18-22, sourced from a trusted online platform for research participant recruitment.

Ultimately, we received 162 survey responses. The results showed a slight preference for AI-generated responses, though concerns about AI’s impact on education and work remain.

Learners See Value in AI Advice

Our survey, encompassing both CareerVillage learners and early adult participants, revealed a preference for AI-generated advice. These results suggest that AI can be a valuable tool for delivering clear and informative responses. However, it is important to note the scope was limited. There were just two survey questions in which respondents directly compared AI-generated and non-AI responses to learner queries, ultimately choosing a preferred piece of advice from two unlabeled options.

Three other questions asked respondents to read a specific piece of advice and rate its helpfulness on a scale of one to five. Of these three survey questions, two asked respondents to rate AI-generated answers to learner queries. Each was rated at an average of 4.5, while the non-AI answer was rated at 4.4 – not a meaningful difference.

Beyond the Hype: Learners Want Ethical Use

When asked an open-ended question about their general feelings toward AI (How do you feel about AI? Do you use it in your education or personal life?), respondents expressed a mix of enthusiasm and concern, especially over the potential for misuse:

• “I think AI can be a useful tool in school, but it shouldn’t be used to cheat. I do use AI to check my work for mistakes or see where I need to improve.”
• “I sometimes use it, but I don’t rely on it. I’ll use it for ideas and stuff.”
• “AI can help assist grammatical errors in work; however, it shouldn’t be taken advantage of nor used to plagiarize to take someone else’s credit.”
• “Love it! Helpful but can be misused”
• “I like it a bit. I like to use it in my education and personal life for small things but not big things. I don’t like that AI is taking over jobs.”

The Road Ahead: Balancing Innovation with Trust

These findings have us re-evaluating how CareerVillage handles AI-generated content. While the survey suggests learners find value in AI, it also raises important questions for our team to consider:

• Should users be allowed to copy-paste AI answers?
• Should AI-generated answers be labeled?
• How can we balance the benefits of AI with genuine human advice?

A Conversation, Not a Conclusion

The survey results are just the beginning of a crucial conversation. The CareerVillage team is committed to striking a balance between embracing innovation and maintaining the authenticity and trust that our community of students, educators, and volunteers thrives on. We’re all in this together, navigating the ever-evolving world of career learning!