Quick answer: Self-host when your data is sensitive, your task is simple, and your volume is under roughly 50,000 items. Hire a managed provider when you need domain experts, tight turnaround, or RLHF at a scale you cannot staff.

Most teams overspend in one direction or the other. I have seen startups sign enterprise contracts for 2,000 images, and I have seen a five-person team try to hand-label 300,000 support tickets in-house over a quarter. Neither ends well.

What are the real options?

Open-source tools such as Label Studio give you the software for free; you supply the annotators. Managed providers supply both software and people.

How do you size the decision?

Start with three numbers: item count, minutes per item, and the hourly rate of whoever labels. If your dataset is 8,000 images at 40 seconds each, that is about 89 hours of work — under three weeks for one person. Doing that in-house is usually cheaper than a vendor minimum.

Now multiply the same math by a 200,000-item request. That is over 2,200 hours, which is where a managed provider’s throughput wins by default.

When do you need domain experts?

This is the question most teams get wrong. Labeling “is this a cat” needs no expertise. Labeling a medical scan, a legal clause, or a financial anomaly does. If your task requires a credential or years of context, general crowd workers will produce confident nonsense.

Managed providers with specialist pools exist for exactly this. It is the strongest argument for paying a premium.

How do you protect data you cannot share?

Regulated data — health records, personal data, proprietary source — often cannot leave your environment. That rules out most hosted vendors unless they offer a deployment model that satisfies your legal team. Self-hosting removes the question entirely.

How do you judge quality before committing?

• Run a pilot of 200-500 items and personally review 50.
• Measure inter-annotator agreement on an overlapping subset.
• Check turnaround time against what was promised, not what was pitched.
• Ask what happens when labels are wrong — rework, credit, or nothing.

The pilot costs almost nothing next to the full contract and it is where vendors differentiate.

What about RLHF and evaluation?

Preference ranking and red-teaming are a different beast. These need workers who can judge model output quality and follow subtle rubrics, and the volume is high. This is the strongest use case for a managed provider like Scale, where the workforce and quality tooling already exist.

Who should self-host: small teams with manageable volume and sensitive data. Who should hire out: anyone needing specialists or RLHF at scale. The mistake to avoid: choosing on price alone — bad labels quietly cost more than good ones ever do.

Quick Summary

• Self-host under roughly 50,000 simple items; hire managed at higher volume.
• Multiply item count by minutes per item before deciding — the number is often surprising.
• Domain-expert tasks need specialist pools, not crowd workers.
• Regulated data usually forces self-hosting.
• Always run a paid pilot and review a sample yourself before scaling.

Quick answer: You upload unlabeled data through the Scale API or dashboard, define a labeling task with instructions, and Scale returns annotated data plus quality metrics. Pricing is quoted per project, not published per item.

This is not a self-serve tool with a free tier you can poke at. It is an enterprise service aimed at teams with serious data needs and budget.

What does Scale AI actually do?

Three main lines: data labeling (images, text, audio, documents), human feedback for RLHF (ranking model outputs, writing preferred answers), and model evaluation (red-teaming and benchmark runs). The common thread is humans in the loop at volume.

How does a labeling project start?

1. Create a project and define the task type in the dashboard.
2. Upload a sample batch of a few hundred items.
3. Write labeling instructions with positive and negative examples.
4. Run a calibration batch and review the returned labels.
5. Adjust instructions, then scale up to the full dataset.

Skipping the calibration batch is the expensive mistake. Instructions that look obvious to you get interpreted five ways by annotators, and you find out only after paying for thousands of bad labels.

How do you integrate via the API?

import requests

resp = requests.post(
    "https://api.scale.com/v1/task",
    auth=("live_yourkey", ""),
    json={"callback_url": "https://yourapp/cb",
          "instruction": "Draw a box around every vehicle",
          "attachment": "https://.../img.jpg",
          "type": "imageannotation"})
print(resp.json()["task_id"])

Results arrive at your callback URL when complete. For bulk work you submit tasks programmatically and track them in the dashboard.

How is pricing structured?

Scale does not publish a rate card. Pricing depends on task complexity, volume, and turnaround. Simple classification is cheaper per item than detailed segmentation; rush jobs cost more. Expect a sales conversation and a per-project quote rather than a sign-up-and-go model.

How does it compare to self-hosting?

If you have a small team and modest data, a free tool like Label Studio covers you. Scale makes sense when volume, turnaround, or specialized expertise exceeds what you can staff.

What are the trade-offs?

• Your data leaves your infrastructure — check the contract for residency and usage terms.
• Cost is opaque until you talk to sales, which slows experimentation.
• The API is solid but batch turnaround is not instant.
• Heavy optimization for specialist tasks may still need your own reviewers.

Where does it genuinely earn its keep?

RLHF at scale is the clearest case. Getting thousands of ranked comparisons from trained contractors, with quality monitoring, is something almost no internal team can staff overnight. The same goes for evaluation datasets that need domain experts rather than generalists.

Who this suits: funded teams with large labeling budgets and tight timelines. Who should skip it: startups and solo builders — the minimum engagement and quote process make it a poor fit until volume justifies it.

Quick Summary

• Scale AI is a managed data platform for labeling, RLHF, and evaluation.
• Workflows run through a dashboard or the Scale API with callbacks.
• Pricing is quoted per project, not published per item.
• Always run a calibration batch before scaling up.
• Best for funded teams at volume; free tools cover small projects.