Skip to main content

Data Lakes

Data Lakes are curated collections of documents that power AI retrieval in Bike4Mind. They provide structured, scoped access to knowledge bases so the AI can search and reference domain-specific content during conversations.

How It Works​

A Data Lake groups files under a common tag prefix and makes them available to AI tools like knowledge base search and retrieval. Each Data Lake has:

  • Name and slug -- a human-readable identifier
  • File tag prefix -- scopes which uploaded files belong to the lake
  • Access controls -- user tags and entitlement gates determine who can query the lake
  • Organization scoping -- lakes can be scoped to a specific organization

Access Control​

Data Lakes support two layers of access control:

  1. User tags -- users with the required tag can access the lake's content
  2. Entitlements -- subscription-level or role-based entitlement keys provide an additional gate

When an AI tool performs a knowledge base search, it automatically resolves which Data Lakes the current user can access based on their tags and entitlements.

Lifecycle​

Data Lakes follow a lifecycle with these statuses:

StatusDescription
DraftBeing configured, not yet active
ActiveAvailable for AI retrieval
ArchivingBeing moved to cold storage
ArchivedIn cold storage, not searchable
UnarchivingBeing brought back from archive
RestoringBeing recovered from a delete
DeletingRemoval in progress

Archiving and deleting are two separate reversal paths, and each has its own in-progress status -- Unarchiving comes back from Archived, Restoring comes back from Deleted. Only one lifecycle action can hold a lake at a time.

A new Data Lake starts as Draft, and a draft lake is excluded from AI retrieval. The assistant cannot find its files even after they finish uploading and indexing. Adding files does not publish a lake -- publishing is a deliberate step, so that no lake starts grounding answers without someone choosing it. Use Publish in the Data Lakes list once the lake is ready.

This holds for a lake the assistant creates from chat with the Save to Data Lake tool (see Smart Tools) too: it starts as Draft, and saving files into it does not publish it.

Use Cases​

  • Domain-specific knowledge -- upload medical literature, legal documents, or technical manuals and scope retrieval to authorized users
  • Organization knowledge bases -- each org can maintain its own curated content
  • Product documentation -- give AI access to internal docs for support workflows

FAQ​

How do I create a Data Lake? Data Lakes are created and managed by administrators. Contact your admin to set up a new lake with the appropriate tag prefix and access controls.

Can I query multiple Data Lakes at once? Yes. The AI retrieval tools automatically search across all Data Lakes you have access to based on your user tags and entitlements.

How are files added to a Data Lake? Files are tagged with the lake's file tag prefix when uploaded. The lake tracks file count and total size automatically.

Can an API key ground a session on a lake its owner is not a member of? Only if the key was bound to that lake when it was minted. Creating a session can name preauthorizedLakeIds to admit a lake the creator manages but is not a member of; for an API-key caller that admission additionally requires the key itself to carry the lake in its binding list. Bindings are set at mint time by an administrator (Admin -> Users -> Generate API Key), can only name lakes the key's owner already manages, and cannot be added to an existing key -- which is what stops a leaked key from reaching every lake its owner manages. A caller whose key lacks the binding gets 403 This API key is not bound to data lake <id>; the fix is a newly minted key, not a permission change.

Why does a reply say "Only part of your library was searched"? Grounded answers scan the library under per-turn bounds -- a candidate-document cap and a chunk budget -- and they skip documents whose embeddings came from a different model than the one running the query. When any of that applies, the reply carries a warning banner, and expanding it lists the specific reasons. The important consequence is what the answer does not prove: a "nothing found" on a partial scan is not evidence that the library holds nothing relevant. Narrow the question, or wait if the banner says documents are still being re-indexed.

I got "this data lake moved to '...' while it was being archived/deleted/restored". What happened? Two lifecycle actions ran on the same lake at once -- for example Archive and Delete were both started from the lake's panel before the first finished. Only one of them can complete, and the other reports this instead of overwriting it. The message names the status that won; the lake is in that state and is safe to act on from there. Re-run the action you wanted if it is still available.

I saved a research configuration and nothing new appeared in the lake's History tab. Why? The History tab records a research-configuration edit only when a setting meaningfully changes. Saving the same values again, or reordering the same entries in Only these sites, Never these sites or Tags to propose, does not add a row.

Who can connect a Google Drive folder to a Data Lake? On an organization Data Lake, an organization owner or manager (or a platform admin). On a personal Data Lake, only the person who created it: the folder syncs using their own linked Google Drive, so unlinking Google Drive from their profile stops that sync until they link it again and reconnect the folder. Either way the Data Lake must be connector-fed and not archived. A curated Data Lake must be switched to connector-fed in its settings first. A Drive folder can feed only one Data Lake, personal or organization. To move a Drive-fed Data Lake into or out of an organization, disconnect its folder first and reconnect it afterwards.

Who can connect a GitHub repository to a Data Lake, and what if the lake is curated? On an organization Data Lake, an organization owner or manager (or a platform admin). On a personal Data Lake, only the person who created it, who is also the only one who can re-sync or disconnect it. The Data Lake must be connector-fed and not archived. A repository can feed only one Data Lake, personal or organization. To move a GitHub-fed Data Lake into or out of an organization, disconnect its repository first and reconnect it afterwards. On a curated Data Lake, Connect GitHub offers to switch it to connector-fed for you. The switch applies to the whole Data Lake, not just GitHub: any connector or scheduled import can then add files to it. It is made only when the connection can actually start, so if it cannot (for example, the lake is archived) the lake stays curated and nothing changes. The switch is recorded in the lake's History tab. If you leave GitHub's page without finishing, the lake stays connector-fed; switch it back in its settings if you no longer want that.

What happens when I disconnect a Google Drive folder? Every file the connection brought into the Data Lake is permanently deleted, along with its extracted text and search entries, and it is removed from any chat it was attached to. This cannot be undone. The confirmation dialog shows how many files will be deleted before you confirm. Files you uploaded by hand are kept.

Why does my Drive connection say "Disconnecting"? The files are removed in the background, so a large folder can take a few minutes. Until that finishes the connection shows Disconnecting and cannot be synced. If it stops making progress, a Retry disconnect button appears.

Why does my Drive connection say "Credential error"? Google Drive access for the connection was revoked or has expired, so syncing has stopped. Click Reconnect Google Drive, approve Google's consent screen, and the folder re-syncs when you return.

Why is a file from my Drive folder or GitHub repository missing from the Data Lake? Sync skips files it cannot ingest, for example a file over the size limit, a file type that cannot be imported (such as a Google Form), or a GitHub file that is not text. When files were skipped, the Data Lake manager shows a chip such as 3 files not ingested to anyone who can manage the lake. Click it to see each file by name with the reason it was skipped. A file leaves the list once it ingests or is removed from the folder or repository. Files skipped because they could not be downloaded, or because the lake owner's storage was full, are retried on a later sync (for Drive, at the latest on its daily full re-sync).