Datahub RLP
The node queries a data endpoint of the Datahub Rheinland-Pfalz through the dataCycle API v4 using an access token, collects all result pages, resolves selected links (JSON-LD) and emits every content as a flat object in dot notation. It is suited for tourist points of interest, events, tours and accommodation that are subsequently mapped to a Smart Data Model.
- Node type
- datahub-rlp
- Package
- @htag/node-red-contrib-hypertegrity-pointofinterest
- Inputs / outputs
- 1 / 1
- Data source
- Datahub Rheinland-Pfalz (dataCycle API v4)
When to use this node
The Datahub Rheinland-Pfalz is the tourism data platform of the German state of Rhineland-Palatinate and runs on the dataCycle software. Contents such as sights, events, hiking trails or restaurants are maintained there as JSON-LD with many links: a place refers to images, addresses, opening hours and classifications, an event to its venue and its series. The Datahub RLP node takes care of fetching, including pagination, retries on errors and the resolution of these links, and delivers a structure that can be processed without deep JSON-LD knowledge.
Typical use cases are:
- Tourist points of interest of a region transferred into an NGSI-LD context broker as
PointOfInterestentities. - Event calendars fetched regularly and loaded incrementally by modification date.
- Data integration of several data endpoints (saved searches or content collections) into an urban data platform.
How the node works
Trigger
An incoming message starts the fetch. Optionally it overrides endpoint, language or filter.
Fetch pages
The node sends POST requests to /api/v4/endpoints/<ID> and follows meta.total and meta.pages until all pages (or the page limit) are reached. When throttled it waits and retries.
Resolve links
Selected paths are passed as the include parameter; remaining @id references are enriched from contents already loaded in the same query.
Flatten
Every content is converted into a flat object with dot notation, for example address.postalCode.
Emit
All contents in one message, or one message per page as a sequence with msg.parts.
Result: An array of flat contents plus metadata in msg.datahub, ready to be mapped to a Smart Data Model.
The node is trigger-based and does not run schedules itself. Combine it with an Inject node. Running queries are cancelled on a deploy or when Node-RED stops, so that an outdated configuration does not keep requesting pages.
Configuration
The edit dialog has four tabs.
Tab "Connection"
| Field | Type | Default | Meaning |
|---|---|---|---|
| Name | Text | empty | Display name of the node in the flow. |
| Base URL | Text (required) | see below | Address of the dataCycle instance. Must start with http:// or https://. Other dataCycle installations can be reached this way as well. |
| Data endpoint | Text (required) | empty | UUID or slug of the content collection or saved search. If you paste a complete API URL from the Datahub interface, it is split automatically into base URL, data endpoint and (if present) token. |
| Language | Text | de | Requested content language, several as a comma-separated list (for example de,en). |
| Access token | Password (credential) | empty | Token of the Datahub user. Stored encrypted in the Node-RED credentials store, sent in the request body as token and additionally as an Authorization: Bearer header. |
| Test connection & load attributes | Button with "Discovery depth" select (1 to 5, default 2) | – | Samples five contents, shows the collection title, number of contents and pages, and discovers all linkable attributes cascading down to the chosen depth. The result appears on the "Linked Data" tab. |
Default values of the URL fields:
Base URL: https://data.rlp-tourismus.de
Tab "Linked Data"
| Field | Type | Default | Meaning |
|---|---|---|---|
Resolve linked contents (includes) | Checkbox tree | empty | Attributes whose linked contents are fetched completely. Selected paths are passed as the API parameter include; parent paths are added automatically (image for image.author). Paths marked with ⚠ link back to the same content type and multiply the data volume. |
| Select all / Clear selection | Buttons | – | Selects all discovered attributes except the recursive ones, or clears the selection. |
| Add path | Text with button | empty | Adds a path manually, for example image.author, if it did not occur in the sample. Letters, digits, _, :, - and dots as separators are allowed. |
Tab "Query"
| Field | Type | Default | Meaning |
|---|---|---|---|
| Page size | Number (1 to 500) | 20 | Number of contents per API request (page[size]). |
| Page limit | Number | 0 | 0 fetches all pages. Any other value limits the number of requested pages. Independently, a fixed upper bound of 2000 pages per run applies. |
| Request delay | Number in milliseconds | 500 | Pause between two page requests to avoid the rate limit of the Datahub. |
| Sorting | Text | empty | API parameter sort, for example dct:created, dct:modified, name or default. A leading minus reverses the direction. |
| Fields | Text | empty | API parameter fields: restricts the delivered attributes. This is the most effective lever against large data volumes. |
| Additional filter | JSON text area | empty | JSON for the API parameter filter, applied on top of the filters stored in the data endpoint. Invalid JSON produces a warning on save and an error at runtime. |
Tab "Output"
| Field | Type | Default | Meaning |
|---|---|---|---|
| Output | Select | "All pages in one message" | collected: the node collects all pages and sends one message. perPage: one message per page as a sequence with msg.parts; only a single page is kept in memory. |
| Separator | Text | . | Character between the path segments of the flat keys. |
| Max. depth | Number (1 to 20) | 8 | Nesting depth beyond which only the reference (@id) is kept. |
| Simplify arrays | Checkbox | enabled | Produces address.postalCode instead of address.0.postalCode when an array has exactly one entry. |
| Resolve references | Checkbox | enabled | Enriches remaining @id references from contents that were already delivered in full within the same query. In "per page" mode this only works within one page. |
| Max. keys/content | Number | 5000 | Safety net: if a content exceeds this number of keys while flattening, it is truncated and a warning is issued. 0 removes the limit. |
Input
Every message triggers the fetch. Three properties can override the configuration for that single message.
| Property | Type | Meaning |
|---|---|---|
msg.payload | any | Triggers the fetch. The content is not evaluated. |
msg.endpointId | string | Overrides the configured data endpoint (UUID or slug). |
msg.language | string | Overrides the configured language. |
msg.filter | object | Merged with the configured additional filter; matching keys from the message win. Suited for incremental fetches, for example {"attribute": {"dct:modified": {"in": {"min": "2026-09-01"}}}}. |
Output
msg.payload is an array of flat objects, one object per content. Nested and linked data appear in dot notation. msg.topic holds the title of the content collection (from meta.collection.name), or the endpoint id as a fallback.
[
{
"@id": "3f2a9c1e-5b7d-4e8a-9c2f-1d4b6e8a0c3f",
"@type.0": "dcls:POI",
"@type.1": "Place",
"name": "Example Castle Ruin",
"description": "Ruin of a 12th-century hilltop castle.",
"dct:modified": "2026-09-28T10:12:00.000+02:00",
"address.streetAddress": "Burgweg 1",
"address.postalCode": "67655",
"address.addressLocality": "Example Town",
"geo.latitude": 49.4447,
"geo.longitude": 7.7689,
"image.contentUrl": "https://data.rlp-tourismus.de/images/example.jpg",
"dc:classification.name": "Castle"
}
]
| Field | Meaning |
|---|---|
@id, @type.n | JSON-LD identifier and types of the content. A @type array with several entries gets indices; @context is not carried over. |
name, description, dct:modified, … | Attributes of the content as delivered by the API. Which attributes exist depends on the content type and on "Fields". |
address.postalCode, geo.latitude, … | Flattened nested objects. Arrays with more than one entry show the index, for example image.0.contentUrl. |
….@id | Reference to a linked content that was not resolved or lies beyond "Max. depth". |
The node additionally sets the following message properties:
| Property | Meaning |
|---|---|
msg.datahub | Query metadata: baseUrl, endpointId, collection, language, include, total, pages, pagesFetched (collected only) or pageNumber (per page only), pageSize, itemCount, resolvedReferences, truncatedItems, truncated (collected only) and outputMode. |
msg.parts | Only in "per page" mode: id, index (from 0), count (planned pages, 0 if unknown), type: "array" and len (page size), so that a Join node can merge the pages into one flat array. |
msg.complete | Only in "per page" mode on the last message: true. |
Example flow
The package ships with an example flow. Import it in Node-RED via Menu → Import → Examples → @htag/node-red-contrib-hypertegrity-pointofinterest → datahub-rlp. Alternatively download datahub-rlp.json and import it via Menu → Import. Enter the token yourself after importing.
Inject
An Inject node triggers the fetch manually or at a fixed interval.
Datahub RLP
The node fetches a data endpoint with page size 50 and page limit 1, resolves images, locations, addresses and classifications and emits one message per page.
Debug
A Debug node shows the flat contents in the sidebar.
Show example flow as JSON
[
{"id": "tab1", "type": "tab", "label": "Datahub RLP", "disabled": false, "info": ""},
{"id": "inject1", "type": "inject", "z": "tab1", "name": "", "props": [{"p": "payload"}], "repeat": "", "crontab": "", "once": false, "onceDelay": 0.1, "topic": "", "payload": "", "payloadType": "date", "x": 120, "y": 140, "wires": [["datahub1"]]},
{"id": "datahub1", "type": "datahub-rlp", "z": "tab1", "name": "", "baseUrl": "https://data.rlp-tourismus.de", "endpointId": "<ENDPOINT-UUID>", "language": "de", "pageSize": "50", "maxPages": "1", "requestDelay": 500, "outputMode": "perPage", "includes": "[\"image\",\"location\",\"location.address\",\"location.geo\",\"dc:classification\"]", "fields": "", "sort": "", "filterJson": "", "separator": ".", "collapseSingleArrays": true, "maxFlattenDepth": "10", "maxKeysPerItem": "10000", "resolveKnownLinks": true, "x": 300, "y": 140, "wires": [["debug1"]]},
{"id": "debug1", "type": "debug", "z": "tab1", "name": "Datahub-RLP data", "active": true, "tosidebar": true, "console": false, "tostatus": false, "complete": "payload", "targetType": "msg", "x": 510, "y": 140, "wires": []}
]
Status and errors
| Indicator | Meaning |
|---|---|
| Blue dot "page 1..." or "3/12 - page 3..." | The named page is being fetched; with progress if the page count is known. A suffix "(try 2/5)" indicates a retry. |
| Yellow ring "rate limited - waiting 12s (try 2/8)" | The Datahub answered with HTTP 429 or 503. The node waits according to Retry-After or with an exponentially growing pause (1.5 s, 3 s, 6 s, … up to 60 s) and tries up to eight times. |
| Blue dot "resolving & flattening..." | References are being resolved and contents flattened ("collected" mode). |
| Blue dot "sent page 2/12 (100 contents)" | A page was sent ("per page" mode). |
| Green dot "240 contents (12 pages)" or "240 contents in 12 messages" | The run is complete. |
| Red ring "invalid filter" | The configured additional filter is not valid JSON. |
| Red ring "invalid config" | Base URL, data endpoint, token or page size are missing or invalid, for example "No access token configured.". |
| Red ring "failed" | The fetch failed for good. Typical messages: "Datahub request failed - HTTP 401: Access token invalid or not authorized for this data endpoint", "HTTP 404: Data endpoint id/slug unknown" or "Datahub request for page 3 failed after 8 attempts. … The Datahub is throttling this token …". |
| Warning "Datahub page 3 attempt 1/5 failed: … - retrying in 2s" | A transient error (timeout after 60 seconds, network error, HTTP 5xx); the node retries up to five times. |
| Warning "Page limit reached - only 50 of 240 contents were fetched." | The page limit ended the fetch before all contents were loaded. |
| Warning "2 content(s) exceeded the limit of 5000 keys and were truncated. …" | The safety net kicked in; select fewer or no recursive links, or restrict "Fields". |
HTTP 400, 401, 403, 404 and 422 abort without retry. Errors are reported via done(error) and can be handled with a Catch node. A run cancelled by a deploy ends without error and without output.
Notes
- Rate limit: The Datahub throttles the number of requests per token and then answers with HTTP 429. The number of requests matters, not the size of the collection. Effective countermeasures: larger pages, a request delay, a page limit, only the links you really need, a restriction via "Fields" and incremental runs via
filter[attribute][dct:modified]. - Recursive links: Paths such as
superEvent.subEventlead from an event to its series and back to all sibling events. A single content can thereby grow to hundreds of thousands of keys. The attribute discovery marks such paths and excludes them from "Select all". - Memory: For large collections the "per page" output is recommended, optionally followed by a Join node or per-page processing.
- Smart Data Model: The node deliberately does not map the contents to a Smart Data Model itself. Mapping to
PointOfInterest,Eventor other models is done downstream in the NGSI-LD Builder. - Personal data: Resolved links may contain personal data, for example
author,publisheror contact details as aPersonwith name, e-mail address or phone number. Check within your flow which of these attributes may be processed further or published, and observe the terms of use of the Datahub Rheinland-Pfalz. - Credentials: The token is stored encrypted and is not part of exported flows. During the connection test a newly typed token is transmitted; otherwise the editor uses the stored token.