Data aggregation · build vs buy
Public Records Aggregation: What to Build and What to Buy
Buy a data provider when you need broad coverage fast and the questions are standard. Build an aggregator when you need specific official sources, your own review step and control over what is stored and who can see it. Either way, design it so each source is isolated: one failing source should never block the others.
01Build or buy
Buy when
- You need broad coverage quickly
- Your questions are standard
- You accept the provider's data shape and terms
Build when
- You need particular official registries
- A person must review the result before it is used
- You must control storage, access and logging
02How such an aggregator is designed
In the person dossier aggregator I built, each official public or private source is queried over its API through its own adapter. The work runs on Celery and Redis across nine queues. A failing source does not block the others, and the reason for the skip is logged, so the dossier says what is missing instead of failing silently.
03Design rules that matter
- One adapter per source, so a change in one registry touches one piece of code.
- Isolation of failures. Partial results are better than none, as long as gaps are visible.
- A review step. Results are assembled into one person card and checked by a human, with manual enrichment where needed.
- Access control and logging on every lookup and every stored report.
- Official sources only. The value is in sources you can name and rely on.
04What to check before you commit
| Question | Why it matters |
|---|---|
| Is each source official and nameable? | You can explain where every fact came from |
| What happens when a source is down? | The dossier should show a gap, not an error or a guess |
| Who can run a lookup and see a report? | Access roles and a log of every request |
| How is stored data protected? | Encrypted reports, scoped access, a retention rule |
| Who reviews the result? | A person confirms the card before it is used |
A provider can answer some of these; an aggregator you own can answer all of them. The deciding factor is rarely the price of a lookup. It is whether you can show, later, how a given fact got into a report and who saw it.
FAQShort answers
- What is a person dossier aggregator?
- A system that collects information about a person from official public and private sources over their APIs, assembles it into one card and keeps it reviewed and access-controlled.
- Why use a queue?
- Sources respond at different speeds and fail independently. Queues let the system work on them in parallel and keep going when one fails.
- Is it better to build or buy?
- Buy for breadth and speed on standard questions; build for specific sources, review and control.
Have a task like this? Write to me.
→Read next
- Approval rules as dataprinciple
- How to integrate an LLM into existing business softwarehow to
- How to hire a freelance AI engineerhow to hire
- Case: person dossier aggregatorrelated page