Sixteen decisions, sixteen diagrams, one running example — how raw records become one clean, searchable shape. By ingoampt.
Sixteen decisions, sixteen diagrams, one running example — how raw records become one clean, searchable shape. By ingoampt.
The Complete Mapping Stage: Every Decision, Visualized
Sixteen decisions, sixteen diagrams, one running example — how raw records become one clean, searchable shape. By ingoampt.
This is the full, detailed walkthrough of the mapping stage — the process that turns preserved raw records into one clean, shared shape a search system can actually use. Every decision below gets its own small example and its own diagram, so you can see the shape of the problem, not just read about it.
We use one small, invented reference-library archive throughout — similar in spirit to a service like PubMed, simplified so the ideas stay clear.
1Design the shared shape first
Before writing any mapping code, decide the destination: a small set of tables whose columns mean the same thing regardless of which record type or source they came from.
2Build the first record type
Pick the single most common type and get it fully working, tested, before touching any other type.
<article id="10.1234/ex.2025.041">
<author>Maria Chen</author>
<title>Adaptive Filtering for Noisy Sensor Streams</title>
<journal>Journal of Applied Signal Processing</journal>
<year>2025</year>
</article>3Prove safety before adding more
Before expanding, prove the machinery is safe: a crash mid-write leaves nothing behind, and mapping the same record twice never creates duplicates.
The red/green cycle used for every new rule
4Add a second type — and see what changes
Choose a second type deliberately close to the first, so you can see exactly what has to change: a conference paper, whose venue comes from a different source field.
Key ideaThe reader of the clean table never needs to know whether the venue originally said journal or booktitle. That difference is absorbed here, once.
5Books and chapters — editors, not just authors
Books introduce a real new idea: some are credited to editors rather than a single author.
<book id="books/ex/Handbook25">
<editor>Priya Anand</editor>
<editor>Tom Field</editor>
<publisher>Example Academic Press</publisher>
<series>Systems Engineering Series</series>
</book>6Theses — an institution, not a venue
A thesis was submitted to a university, not published in competition with other work. Forcing the university name into venue would misrepresent what kind of document it is.
7Containers and datasets — not everything is an article
A proceedings volume holds many papers; it is not itself a paper. A dataset entry describes data, not an argument. Both need their own candidate_role so they never crowd out real publications in search.
8Fine bibliographic detail
Once the basic shape works everywhere, add finer citation detail: volume, issue, page range — copied exactly, never reformatted.
9Links and a conservative identifier
An electronic link sometimes contains, in disguise, a resolvable identifier. Recognise it — but only on a complete, unambiguous match.
10A validated identifier — ISBNs
ISBNs carry a built-in checksum, so the mapping stage can validate, not just copy.
11Relations between records
Some fields point to a different record entirely — a cross-reference from a paper to the proceedings volume it belongs to.
12The remaining link types
A few smaller link kinds round out the picture: archive discs, streamed recordings, and links attached to the publisher or series name.
13Identity records — not everything is a publication
Some records exist only to identify a person — a homepage entry, not a paper. Treated naively, the name field would falsely become “authorship.”
<www id="homepages/44/Chen-M">
<author>Maria Chen</author>
<title>Home Page</title>
</www>14Notes — one field, many meanings
The same field name carries wildly different meanings depending on context — an affiliation here, a publication status there.
15Fields that stay source-only, on purpose
Some fields are too ambiguous to map safely yet. The correct move is to wait, not guess.
16Everything, in one picture
| # | Decision | What it adds |
|---|---|---|
| 1 | Shared shape | Seven destination tables, agreed in advance |
| 2 | First type | Title, year, authors, venue — one type, working end to end |
| 3 | Prove safety | Rollback and duplicate protection, tested before scaling up |
| 4 | Second type | Venue from a different source field, same clean column |
| 5 | Books/chapters | Editors kept distinct from authors; publisher, series |
| 6 | Theses | A separate institution column, never disguised as venue |
| 7 | Containers/datasets | Candidate roles beyond “publication” |
| 8 | Fine detail | Volume, issue, pages — copied exactly |
| 9 | Links + identifiers | DOI recognised only on an exact, complete match |
| 10 | Validated identifiers | ISBN checksum validation |
| 11 | Relations | Pointers to other records, resolved later |
| 12 | Remaining links | Archive, recording, publisher/series links |
| 13 | Identity records | Homepages kept honest, excluded from publication search |
| 14 | Notes | Structured but uninterpreted free text |
| 15 | Source-only fields | Ambiguous fields left unmapped until a safe rule exists |
| 16 | Result | One clean layer, ready for indexing and search |
FAQ
- Why so many small decisions instead of one big function?
- Each is independently testable and independently risky. A single giant function hides which rule broke when something goes wrong; sixteen small ones make every failure traceable to one exact cause.
- Why do containers and datasets get their own role instead of being excluded?
- They are legitimate, useful records — just not publications. A separate role means they stay searchable on purpose, without contaminating ordinary publication search.
- Isn’t leaving fields “source-only” just delaying work?
- It delays work you are not yet ready to do safely. A wrong guess baked into millions of records costs far more to fix than a field that simply waits for its proper rule.
- Why validate ISBNs instead of just trusting the source?
- A checksum is a free, mechanical way to know whether a value is trustworthy before relying on it for an exact match.
- What’s the payoff for all sixteen steps?
- A clean layer where every column means exactly one thing, every value is provably faithful, and every missing value is honestly missing — the foundation a fast, trustworthy search system needs.
This article was written by ingoampt.
