How the Vocal Team Selects Challenge Winners

by Jessica Wharton about a month ago in Challenges · updated about a month ago

Offering transparency and clarity into every phase of the judging process.

This bit of clarification has been a long time coming now. The Little Black Book Challenge closed around the middle of March of 2021, and it proved so much larger than we ever could have anticipated. I decided right then, as soon as it really hit us just how huge the impact of that Challenge had been, that it was time to offer more information and transparency about how Challenges actually operate. We knew that we had more new creators coming on board than we had ever seen at once, and we knew that with a prize that large, the stakes were a lot higher. It was time for an update.

So, I started writing this story to explain exactly how our team evaluates Challenge submissions—what our workflow is like internally, what we look for, who gets the final say. I spent a few extra moments here and there each day, adding to it when I got the chance, hoping to publish it shortly after announcing the winners of that Challenge. And then everything else got in the way. The absolute explosion of interest in Challenges that followed also meant intense demand on our moderation and customer support teams, so I spent time ensuring that we adequately prepared those growing teams; we began focusing on development and new features, getting Subscribe ready for you, working on the upkeep of existing safety features and protocols, and as a result, I regrettably put this story on the back burner.

Since March it's sat in my drafts, making its way from one day's to-do list to the next. It was something that I wanted to do, but not something for which I was really accountable to anyone. That is, no one was sitting around anticipating this story, so if it took me a few months to finish and publish, no one would know the difference. If there were interruptions to the platform, delays, or security issues, then those would all be things that would impact how you use Vocal—so they had to take priority, and in the meantime, I figured, this story wouldn't be missed.

But as it turns out, this story was missed. My inclination that the more Challenges grew, the more transparency we would need to provide was more accurate than I may have even anticipated. So now, I am accountable for this story, to everyone hoping for clarity. I am glad that I'll be able to take this chance to answer common questions and concerns, and I'm sorry that I wasn't able to do so sooner.

We want to add some transparency around the judging process, but first, I want to thank each of you who reached out to provide genuine feedback about Challenges and our selection process. Vocal is growing at an exponential rate now, and comparatively, we remain a relatively lean team. Many of us who have been here since the very early days of Vocal are still the ones reading your inquiries, working late into the night on new feature development, and writing email communications that you see from us every week. We love to do it, and we do it all for you—but it does mean that sometimes we lose sight of the details and intricacies of what we've already created. So when something happens that causes an influx of creators to contact us and make a constructive point, it's not only beneficial, it's incredibly grounding. It forces us to take a step back and review all of the things we've built and are still in the process of building.

I don’t know that Vocal will ever run exactly like a well-oiled machine, because, well, we're not a machine. We're a group of people in an office who work together to try to build something that works for you. In short, we’re learning all the time—what’s important to you, what you’d like to see, how you’d change Vocal if you were in charge—and feedback from you teaches us more than anything.

How judging works: Phase 1

So, what does the Challenge judging process look like here at Vocal? First, it's important to know that our team of Moderators are educated in content curation as a part of their moderation training process. They're the ones who select Top Stories, choose creators to Spotlight, and even nominate content that’s featured on our social media. So, as they review submissions for publication, they begin by doing a preliminary round of selection. Anything that jumps out as above average from a first impression is highlighted during this round, but this is also where stories are sent back to creators and denied publication if they don't meet our Community Guidelines.

If you're hoping to do well in a Challenge, it's crucial to keep in mind that moderators are looking for content that stands out here. It could be masterful use of language, a beautifully written first or last line, or an extraordinary take on the prompt that sets your story apart from the rest, but if there’s not something about it that makes it truly unique, the entry is unlikely to make it past this first round, even if it's a perfectly fine story. It's important to keep in mind that moderators see hundreds or thousands of entries for most Challenges, and in most cases, that means reading hundreds or thousands of entries that are very similar in how they interpret the prompt.

Phase 2

In the next step in our judging process, we moderate all stories entered into a Challenge a second time. In Phase 1, we have a queue where moderators review all submissions, whether the stories are Challenge entries or just stories submitted to a Community. We ensure they meet our Community Guidelines, and if so, we publish them on the platform. Then, all stories entered into Challenges come into a separate review queue once they're published. This time, we're checking specifically to make sure that stories meet the Challenge's prompt as well, instead of just our publication guidelines. Phase Two is when a story is removed from a Challenge if it does not meet the criteria but will remain published on Vocal.

The moderators here are basically looking to surface stories with the same qualities as in the first phase of review—Phase 2 just ensures that all entries are reviewed more than once, so that nothing extraordinary falls through the cracks.

Also note that our Moderation Team may complete these phases of review from the back end—meaning a read wouldn’t be triggered on your story, since we’re viewing it from our internal database and not the story link. If you don't see that your story was read in your Stats, it doesn't mean your story wasn't reviewed for consideration—just that it was likely reviewed in these preliminary phases from our side of Vocal.

Phase 3

After this, we usually end up with a first longlist of one- to several-hundred stories, generally somewhere around 30% of total entries depending on the Challenge, that made their way through the first 2 phases of judging. Then, a few of our more senior content members of the Vocal team (all of whom have extensive writing and literature backgrounds, as well as years of moderation and curation experience at Vocal) go through the list of a few hundred and narrow it to a final shortlist somewhere between 10 to 50. Phase 3 is a collaborative effort, to make sure that different tastes and viewpoints are taken into account.

Phase 4

At this point, the final shortlist moves on to the judges of the Challenge.

For most Challenges, whether branded or not, the panel includes Erica Wagner, who’s a professional literary judge we’re lucky enough to have as a member of the Vocal team. For larger-scale or branded Challenges, the additional judges are hand-selected, sometimes by a brand, or perhaps are themselves representative of a brand. For smaller-scale or non-branded Challenges that don't have an external judge, we’ll include someone in upper management at Vocal (like myself!) to accompany Erica, instead of an external brand rep or topic specialist.

The judges will usually go through and score the shortlist of stories on a scale of either 1-5 or 1-10 independently. This is the way we’ve found we get the most impartial scores (it's better than asking judges to rank their favorites in order, because we can account better for stories that absolutely blow each individual judge away). Often, judges have very different tastes, so this helps us take that into consideration; then, once all judges have graded all shortlisted entries, all the scores are totaled from there. We then place the stories with the three highest score totals first, second, and third, respectively, and we break any ties between places that the judges can't reconcile via community engagement—that is, Likes and Reads that the entries have received.

I hope this helps shed a bit more light on what goes into hosting, running, and judging Challenges for those of you seeking extra clarification. From our team here at Vocal to you creators, I want to reiterate that we always hear you and we appreciate your feedback. You help us evolve Vocal every day into a better iteration of itself. Thank you for learning and growing with us.

FAQ

There are only a couple judges specified in a Challenge, and thousands of entries. How is there time to actually review every single entry?

This was a question we saw frequently asked particularly at the end of the Little Black Book Challenge. There were over 12,000 entries and only two weeks for the judges to read them - there's no way that each entry was actually reviewed, right?!

But we promise that every single entry is reviewed! It's important to keep in mind that the first two phases of judging occur simultaneous to and immediately following the moderation process. This means that stories are reviewed throughout the duration of the Challenge and are surfaced constantly; we don't wait until the close of the Challenge to begin judging.

Also keep in mind that we already review thousands of stories submitted to Vocal each day! We're prepared to handle influxes of entries to a Challenge, because we have similar workflows happening every single day.

I don't think that the stories chosen were very well written. Why were they picked?

For most Challenges on Vocal, it's important to remember that Challenges are presented as storytelling contests over writing contests. Some of our creators are professional writers, authors, bloggers—and many of our creators have very little writing experience at all. They just want to share stories, and that's what Vocal was built for.

Our judges take how the creator conveys the story into account—and often, this does mean appraising writing ability and quality, especially in creative writing Challenges! But the judges also consider how gripping, moving, or beautiful the story is, and if there's a typo or a run-on sentence in the story that our judges love the most, and it doesn't detract from the overall impact of the entry, they just may decide that it's incredible enough to overlook technical errors. A typo or grammar issue may be the deciding factor between two stories that are neck-and-neck, or may bump an entry to second or third pace that would otherwise take the top spot—but for a story that absolutely blows the judges away, they won't disqualify you from consideration for a small mistake.

How strict is the word count requirement?

The above answer goes for everything, really—our team considers every published story that's entered into a Challenge, and sometimes they might decide that technical errors or guidelines are just outweighed by how truly awesome a story is. The guidelines and requirements are strict, but as long as you consider them within reason, they won't disqualify you from consideration right away.

Our quality and safety-focused Community Guidelines serve as a first line of defense; other guidelines are intended to make expectations clear. Our judges can't adequately compare a 10,000 word novella to a 1,000 word flash fiction piece, so our Challenge guidelines are outlined on each Challenge page so that each entrant knows what to expect. We want you to understand what we're asking for and anticipate seeing in your entries. So, keep in mind that if your entry doesn't adhere to Challenge guidelines, it very well could hurt your chances of winning, even if it doesn't automatically disqualify you. Judges often love stories that are entered, but make final decisions in favor of other entries because they didn't quite meet the guidelines provided, and others did. It takes a truly amazing story to make our judges overlook certain mistakes, but sometimes stories are just that great.

Not to mention, you should always keep in mind that if a story doesn't meet publication guidelines, it won't be approved by our moderators, and won't even make its way to be evaluated for a Challenge—so stories under 600 words, for example, won't make their way into a Challenge for consideration in the first place.

Does a Challenge being sponsored by a brand affect the outcome?

In a branded Challenge, you have a whole brand to impress in addition to the Vocal judges. Brand-affiliated judges usually look for entries that align with their brand ethics and values, and they often look forward to re-sharing the winning stories with their consumer audiences. So a family-friendly brand may not be interested in a hilarious entry that leans more PG-13, or an animal-loving brand may not love a submission about hunting. It's all relative—we always recommend looking into our brand partners to make sure you understand them as brands before crafting your entry for partnered Challenges.

How do I know if my story has been approved for a challenge?

You can confirm you've successfully submitted a story to a Challenge by visiting your Stories, where the Challenge entered will appear directly next to your submitted draft; or, you can go directly to the Challenge page, and you'll see any entries submitted by your account listed under “My Submissions” towards the bottom of the page. If your story was removed from a Challenge, you'll receive an email alerting you and have the chance to edit it to be more applicable to the contest prompt.

I missed the deadline - can the judges still consider my story?

Unfortunately, no. Once the Challenge’s deadline has passed, the platform no longer accepts submissions, so you won't be able to enter your story for consideration after it closes. However, we encourage you to proceed with publishing your story, as you'll still be able to earn from it and accept tips from your readers.

How does entering a Challenge more than once affect my chances?

There are no rules against entering a Challenge multiple times—as long as the entries follow the Challenge prompt, no entries will be disqualified simply because you've entered before. The more entries you submit, the higher the chances that one of them catches our judges' attention, so feel free to enter away!

Alternatively, you can also enter an entire series of stories to a Challenge, or enter a single entry that's a part of a larger series outside of the Challenge. If you choose to do so, keep in mind that each story will be evaluated independently, not as a part of the series itself. You can do whatever you'd like for your own content and profile to challenge yourself, but know that our judges will be reading each story as a standalone entry, so it should be able to hold its own without relying on any corresponding parts of the series considered.

Take for example this third place winner of the Doomsday Diary Challenge. This winning story ended up being the first part in a series that's still ongoing, even two months later. But on its own, as a standalone entry, it told a complete story, and the judges loved it.

Challenges

Jessica Wharton

Head of Creator Experience @ Vocal

Receive stories by Jessica Wharton in your feed
Jessica Wharton
Read next: How to Keep Your Creativity Alive During Quarantine

Find us on socal media

Miscellaneous links