HIVE-29706: Add AGENTS.md, SECURITY.md, and THREAT_MODEL.md#6535
Conversation
… discoverability v0 threat model produced by the ASF Security team via threat-model-producer (Michael Scovetta rubric, run with Claude Opus) for the PMC to review, correct, and own. Wires the AGENTS.md -> SECURITY.md -> THREAT_MODEL.md discoverability chain the scan agent follows. Every non-trivial claim is provenance-tagged; open questions for the PMC are collected in THREAT_MODEL.md section 14. Generated-by: Claude Opus 4.8 (1M context)
|
Thank you! I will check the draft |
|
Thanks @okumin — no rush. The most useful read is the §14 "Open questions" section at the end; those are where I inferred a position and would value your confirmation or correction. |
okumin
left a comment
okumin
left a comment
There was a problem hiding this comment.
I answered some obvious points. I'm still checking the remaining
Incorporates okumin's PR apache#6535 review: - direct Hive Metastore access in scope (HMS enforces caller authz at the application level; Spark et al. connect directly) -> §3.3/§4/§11a - UDF/SerDe/TRANSFORM code-execution detail: built-in UDF blacklist (reflect/reflect2/java_method/in_file), custom UDF/SerDe admin trust, TRANSFORM disable via DisallowTransformHook -> §7/§8/§11a - §14 Q1/Q2 promoted to maintainer; Q7/Q9/Q12 annotated PMC-reviewing Generated-by: Claude Opus 4.8
|
Thanks
I've left these as "PMC reviewing" in §14 pending your follow-up, so nothing's prematurely locked:
On your off-topic question — separate THREAT_MODEL.md for the Metastore: my lean is to keep a single THREAT_MODEL.md but with clearly separated HS2 vs HMS scope/boundary/property subsections, since it's one repo and one discoverability chain (AGENTS.md -> SECURITY.md -> THREAT_MODEL.md) and the triage dispositions are shared. If you'd rather split them, that works too — we'd just point SECURITY.md at both files. Your call; happy to restructure either way. |
asf-ci-hive
added
tests pending
tests passed
and removed
tests unstable
tests pending
labels
…to separate threat models (different security models/params)
|
Thanks @okumin — really useful detail. Your answers are folded into
On your off-topic question — splitting HS2 and HMS into separate threat models: I think that's a good idea given they have genuinely different security models and parameters, and it also reads cleaner for the scan agent. I've noted it as §14 Q15; happy to split this into No rush on the rest of your review — ping me when you've worked through the remaining points and I'll fold them in one pass. |
| isolation, UDF vetting). Anything mis-assigned? | ||
| 10. Confirm the by-design non-guarantees in §7. | ||
| 11. Is super-linear resource use / a hang on a pathological query a bug, or is | ||
| bounding it the operator's job (YARN queues / HS2 limits)? |
There was a problem hiding this comment.
In general, this is an operator responsibility rather than a Hive bug. Hive accepts arbitrary HiveQL, so operators are expected to use HiveServer2 limits (e.g., hive.query.max.length) and YARN resource pools to bound the impact of pathological queries.
If a stronger isolation than HS2 and YARN can provide is required, operators should use separate HS2 instances or separate Hadoop/YARN clusters.
| code-execution-by-design (not a sandbox), per §7?~~ | ||
| 3. Confirm the assumed deployment: clustered, behind an operator-controlled | ||
| perimeter, with Hadoop + a metastore RDBMS + (Ranger or SQL-std auth) + KDC | ||
| as trusted dependencies. |
There was a problem hiding this comment.
Yes. We assume a clustered Hive deployment behind an operator-controlled perimeter. Hadoop, the metastore RDBMS, the configured authorization provider, and the Kerberos KDC are trusted dependencies.
The authorization provider is typically Ranger in production deployments, though SQL-standard authorization may also be used.
| 4. Is the in-scope adversary "a SQL client at the HS2 boundary" (+ a network | ||
| MITM where TLS is off)? Anything to add? | ||
| 5. Confirm operators with storage/metastore-DB/cluster-process access, and | ||
| trusted admins doing authorized actions, are out of model. |
There was a problem hiding this comment.
Yes. Operators with direct storage, metastore DB, or cluster-process access are considered trusted and out of scope. Authorized actions by trusted administrators are also out of model. This threat model focuses on behavior via Hive’s supported interfaces, assuming that the underlying infrastructure and its administrators are trusted.
| perimeter, with Hadoop + a metastore RDBMS + (Ranger or SQL-std auth) + KDC | ||
| as trusted dependencies. | ||
| 4. Is the in-scope adversary "a SQL client at the HS2 boundary" (+ a network | ||
| MITM where TLS is off)? Anything to add? |
There was a problem hiding this comment.
Yes. The primary in-scope adversaries are untrusted clients at Hive service boundaries: SQL clients submitting statements to HS2, and clients accessing the Hive Metastore through supported APIs.
A network MITM is also in scope when TLS or equivalent transport protection is enabled, and a Hive operator is responsible for setting up TLS properly.
|
Thanks @okumin — this is exactly the maintainer input the §14 open questions were fishing for, and it sharpens the model a lot. How I'll fold it in: Answered questions → maintainer-ratified. Your adversary model (untrusted SQL/metastore clients; MITM when TLS is enabled), trusted dependencies (Hadoop, metastore RDBMS, the authz provider, KDC), and trusted-admin-out-of-scope will move from (inferred) to (maintainer) in the next push. UDF / SerDe / TRANSFORM. The built-in-UDF blacklist ( Metastore direct access (line 186). Agreed it belongs in scope — I'll add direct Hive Metastore access (e.g. from Spark) as an in-scope interface and cite your gist. On a separate Still open (no rush), left as open questions pending your word:
I'll push the fold-in and re-request your review. Thanks again. |
|
Thanks.
Sure. We would like to follow the best practice. Let's go with your suggestion now. I'm still checking the unanswered questions. Wait a while |
§3/§4/§7 adversaries, trusted deps, query-resource bounding; §14 Q3/Q4/Q5/Q11 marked answered. Generated-by: Claude Code (Claude Opus 4.7)
|
@okumin — pushed the fold-in (c8ca131): your 2026-06-27 answers on the adversary model, trusted dependencies, out-of-scope operators/admins, and query-resource bounding are now in the model (§3/§4/§7), and §14 Q3/Q4/Q5/Q11 are marked answered. Still open whenever you have a moment: the |
| admins doing authorized actions, are out of model.~~ | ||
|
|
||
| **Wave 2 — trust boundaries & auth** | ||
| 6. At the client→HS2 boundary, are SQL text, JDBC connection properties, and |
There was a problem hiding this comment.
From the point of view of HS2, SQL or connection parameters are untrusted. Hive can reject non-acceptable operations through an authz plugin or by configuring deny lists (i.e., hive.conf.restricted.list, hive.conf.locked.list, and hive.conf.hidden.list)
| 12. *(PMC reviewing — okumin is checking the exact TLS configuration parameter | ||
| names on the Hive side; §8 TLS lever left unnamed pending that.)* Confirm | ||
| the real names + shipped defaults of the §8 levers (especially | ||
| `hive.server2.authentication` and the default authorization model). |
There was a problem hiding this comment.
Additionally, all credentials or secrets should be managed by Hadoop CredentialProviderAPI. Therefore, Hive administrators should store them in a file or something, and they have to configure hadoop.security.credential.provider.path.
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html
Apache Hive does not guarantee that the content in hive-site.xml or environment variables is never exposed.
There was a problem hiding this comment.
HiveServer2
- To enable authentication,
hive.server2.authenticationand more-specific authentication parameters (typically,hive.server2.authentication.*) need to be configured properly - To enable authorization,
hive.security.authorization.enabled=trueandhive.security.authorization.managerneed to be configured, typically using Apache Ranger.hive.security.authenticator.managershould beorg.apache.hadoop.hive.ql.security.SessionStateUserAuthenticatorto authorize as the HS2 session user. With Ranger,hive.server2.enable.doAsis typically configured tofalse - To enable TLS,
hive.server2.use.SSL=trueand more-specific TLS parameters need to be configured - A Hive administrator should ensure security-related configurations are protected using the appropriate
hive.conf.restricted.list,hive.conf.locked.list, andhive.conf.hidden.listsettings, so untrusted users cannot update sensitive settings or view secret values
You can find some more information here: https://hive.apache.org/docs/latest/admin/setting-up-hiveserver2/#authenticationsecurity-configuration
HiveServer2 Web UI
If the UI port (default: 10002) is exposed to untrusted users, a Hive administrator should configure the following parameters
- To enable authentication, at least one of
hive.server2.webui.use.spnego=true,hive.server2.webui.use.pam=true, orhive.server2.webui.auth.method=LDAP, and more-specific authentication parameters need to be configured - To enable TLS,
hive.server2.webui.use.ssl=trueand more-specific TLS parameters need to be configured
There was a problem hiding this comment.
Hive Metastore
If a Hive administrator exposes Hive Metastore to untrusted users or systems, such as Spark or Flink, they have to configure the following parameters.
- To enable authentication,
metastore.authentication(e.g.,KERBEROSorLDAP) and more-specific authentication parameters (e.g.,metastore.sasl.enabled,metastore.kerberos.*ormetastore.authentication.*) need to be configured properly - To enable authorization,
hive.security.authorization.managershould be configured properly (typically, RangerHiveAuthorizerFactory) andmetastore.pre.event.listenersshould beorg.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.metastore.server.filter.enabled=trueandmetastore.filter.hook=org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizerare also needed - To enable TLS,
metastore.use.SSLand more-specific TLS parameters need to be configured
| 6. At the client→HS2 boundary, are SQL text, JDBC connection properties, and | ||
| session-config overrides all treated as untrusted (subject to the conf | ||
| whitelist)? | ||
| 7. *(PMC reviewing — okumin expects `hive.server2.enable.doAs=false` is the |
There was a problem hiding this comment.
As our threat model recommends using an authz plugin, we can say hive.server2.enable.doAs=false is typical
| 12. *(PMC reviewing — okumin is checking the exact TLS configuration parameter | ||
| names on the Hive side; §8 TLS lever left unnamed pending that.)* Confirm | ||
| the real names + shipped defaults of the §8 levers (especially | ||
| `hive.server2.authentication` and the default authorization model). |
There was a problem hiding this comment.
HiveServer2
- To enable authentication,
hive.server2.authenticationand more-specific authentication parameters (typically,hive.server2.authentication.*) need to be configured properly - To enable authorization,
hive.security.authorization.enabled=trueandhive.security.authorization.managerneed to be configured, typically using Apache Ranger.hive.security.authenticator.managershould beorg.apache.hadoop.hive.ql.security.SessionStateUserAuthenticatorto authorize as the HS2 session user. With Ranger,hive.server2.enable.doAsis typically configured tofalse - To enable TLS,
hive.server2.use.SSL=trueand more-specific TLS parameters need to be configured - A Hive administrator should ensure security-related configurations are protected using the appropriate
hive.conf.restricted.list,hive.conf.locked.list, andhive.conf.hidden.listsettings, so untrusted users cannot update sensitive settings or view secret values
You can find some more information here: https://hive.apache.org/docs/latest/admin/setting-up-hiveserver2/#authenticationsecurity-configuration
HiveServer2 Web UI
If the UI port (default: 10002) is exposed to untrusted users, a Hive administrator should configure the following parameters
- To enable authentication, at least one of
hive.server2.webui.use.spnego=true,hive.server2.webui.use.pam=true, orhive.server2.webui.auth.method=LDAP, and more-specific authentication parameters need to be configured - To enable TLS,
hive.server2.webui.use.ssl=trueand more-specific TLS parameters need to be configured
| 12. *(PMC reviewing — okumin is checking the exact TLS configuration parameter | ||
| names on the Hive side; §8 TLS lever left unnamed pending that.)* Confirm | ||
| the real names + shipped defaults of the §8 levers (especially | ||
| `hive.server2.authentication` and the default authorization model). |
There was a problem hiding this comment.
Hive Metastore
If a Hive administrator exposes Hive Metastore to untrusted users or systems, such as Spark or Flink, they have to configure the following parameters.
- To enable authentication,
metastore.authentication(e.g.,KERBEROSorLDAP) and more-specific authentication parameters (e.g.,metastore.sasl.enabled,metastore.kerberos.*ormetastore.authentication.*) need to be configured properly - To enable authorization,
hive.security.authorization.managershould be configured properly (typically, RangerHiveAuthorizerFactory) andmetastore.pre.event.listenersshould beorg.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.metastore.server.filter.enabled=trueandmetastore.filter.hook=org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizerare also needed - To enable TLS,
metastore.use.SSLand more-specific TLS parameters need to be configured
| intended posture, since HiveServer2 can enforce policies itself; asked for a | ||
| second PMC member to double-check before we finalize.)* Which `doAs` posture | ||
| is the supported/recommended one, and how does it change the authorization | ||
| story? |
There was a problem hiding this comment.
With Apache Ranger, hive.server2.enable.doAs=false is the expected configuration.
| second PMC member to double-check before we finalize.)* Which `doAs` posture | ||
| is the supported/recommended one, and how does it change the authorization | ||
| story? | ||
| 8. What properties does Hive claim to uphold given valid input (auth, authz |
There was a problem hiding this comment.
Given valid input and a secure configuration, Hive is expected to uphold the following properties:
- Authentication: HiveServer2 and Hive Metastore can require clients to authenticate using the configured mechanism, such as Kerberos and LDAP, depending on the endpoint.
- Authorization scoping: Requests are authorized as the authenticated Hive session/metastore user through the configured authorization manager, typically Apache Ranger. Authorization decisions are scoped to Hive objects and operations such as databases, tables, partitions, columns, functions, and relevant metadata operations.
- Metastore authorization: When Hive Metastore is directly exposed to clients, metastore-side authorization can be enforced through
HiveMetaStoreAuthorizeras a pre-event listener, and metadata read/list results can be filtered server-side using the metastore filter hook. - Transport protection: HiveServer2, Hive Metastore, and the HiveServer2 Web UI can protect traffic confidentiality and integrity when TLS/SSL is enabled and correctly configured.
- Configuration protection: Sensitive or security-critical configuration values can be restricted, locked, or hidden so that untrusted users cannot change security posture at runtime or view secret values.
- Credential handling: Secrets can be externalized through credential providers instead of being stored directly in cleartext configuration files.
These properties are configuration-dependent. Hive does not by itself guarantee authorization for users who bypass Hive and access the underlying storage or metastore database directly; those paths must be protected separately by HDFS/object-store/IAM/database controls.
| than the network level (reflected in §3.3/§4), and is still considering | ||
| whether to treat Ranger as the only supported authorization system.)* | ||
| Confirm the operator-owned list in §6 (TLS, authz-model choice, network | ||
| isolation, UDF vetting). Anything mis-assigned? |
There was a problem hiding this comment.
Let's say Ranger is the primary authorization plugin
| whether to treat Ranger as the only supported authorization system.)* | ||
| Confirm the operator-owned list in §6 (TLS, authz-model choice, network | ||
| isolation, UDF vetting). Anything mis-assigned? | ||
| 10. Confirm the by-design non-guarantees in §7. |
| the real names + shipped defaults of the §8 levers (especially | ||
| `hive.server2.authentication` and the default authorization model). | ||
| 13. What do scanners/fuzzers/researchers most often report that you consider a | ||
| non-finding? (Feeds §11a.) |
There was a problem hiding this comment.
- Secret exposure when a Hive administrator stores secrets in
hive-site.xml,metastore-site.xml, environment variables, or system properties. They should use Hadoop CredentialProviderAPI instead
| 14. Hive has no in-repo `SECURITY.md`/`THREAT_MODEL.md` today; this PR adds | ||
| them and wires `AGENTS.md → SECURITY.md → THREAT_MODEL.md`. Confirm this | ||
| in-repo model is canonical (vs the cwiki security pages), how it should | ||
| reference those pages, and who owns revisions. |
There was a problem hiding this comment.
As of today, we may say this is the canonical page. We have some security information scattered across our website. However, I can't guarantee they are up to date.
| `THREAT_MODEL.md` (e.g. `THREAT_MODEL.md` for HS2 + a separate one for HMS, | ||
| each discoverable), rather than one combined document? Leaning toward | ||
| splitting if it makes the per-component scope cleaner for both triagers and | ||
| scan agents; the PMC's call on structure. |
There was a problem hiding this comment.
I'm biased toward this statement and want to follow it.
my suggestion is to keep one file but split it into clearly-labelled HiveServer2 and Hive Metastore sections, each with its own scope / adversary / trust-boundary subsection — rather than two files
| ### Key scoping facts (see THREAT_MODEL.md) | ||
| - The **HiveServer2** SQL front door is the primary untrusted boundary; the | ||
| Metastore and execution cluster are assumed to run inside an | ||
| operator-controlled perimeter. |
There was a problem hiding this comment.
Should we update as follows in the end?
The HiveServer2 SQL front door and directly exposed Hive Metastore endpoints are the primary untrusted boundaries; execution clusters and internal service dependencies are assumed to run inside an operator-controlled perimeter.
Apply the Hive PMC's (okumin) answers to the model's open questions: promote the resolved §4/§5/§6/§7 claims from (inferred) to (maintainer); reorganize §8 into HiveServer2 / Web UI / Metastore / Credentials blocks with the maintainer-supplied parameter names; add the secret-exposure known non-finding to §11a; record the in-repo model as canonical. Also update AGENTS.md's primary-boundary line per the PMC's wording. Generated-by: Claude Code (Claude Opus 4.8)
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
| `hive-site.xml` / `metastore-site.xml` / env vars / system properties | ||
| instead of using the Hadoop `CredentialProviderAPI`. Folded into §11a.)* | ||
| ~~What do scanners/fuzzers/researchers most often report that you consider a | ||
| non-finding?~~ |
There was a problem hiding this comment.
We may add the following sentence to §11a.
Hive defaults are not a complete secure-production baseline; some defaults
prioritize ease of adoption, compatibility, or trusted/local deployments.
Operators are responsible for enabling and correctly configuring authentication,
authorization, transport security, network isolation, and related controls for
internet-exposed or multi-tenant production deployments. `OUT-OF-MODEL: operator responsibility`
We sometimes receive a security report with default but insecure configurations.
Co-authored-by: Shohei Okumiya <okumin@apache.org>
okumin
changed the title
|
I approved and linked HIVE-29706. Based on our commit guideline, I will merged this in 24 hours |
asf-ci-hive
added
tests failed
tests pending
and removed
tests pending
tests failed
labels
|
|
Merged. @potiuk Thanks for your kind support! |
3 participants
This adds a v0 security threat model + discoverability wiring to
apache/hive, produced by the ASF Security team for the Hive PMC to review and own — the pre-flight step for the Glasswing security scan the PMC opted into.What's here
THREAT_MODEL.md— a v0 model (Michael Scovetta rubric, run with Claude Opus) covering the HiveServer2 SQL front door, the Metastore, and the UDF / SerDe / execution layer: trust boundaries, in/out-of-scope adversaries, what Hive upholds vs. what it leaves to the operator (TLS, authorization-model choice, network isolation, UDF vetting), known non-findings, and triage dispositions. Every non-trivial claim is provenance-tagged(documented)/(maintainer)/(inferred); the(inferred)ones are our hypotheses.SECURITY.md— private reporting viasecurity@hive.apache.org+ a pointer to the model.AGENTS.md— wiresAGENTS.md → SECURITY.md → THREAT_MODEL.mdso the scan agent (and researchers) can mechanically find the model.How to engage — this is a draft to react to, not a finished artifact.
THREAT_MODEL.md§14 collects open questions in waves; answer inline a few at a time, correct anything wrong, and the model becomes the PMC's. Once you're happy, we queue the scan in OSS-criticality order. No deadline pressure with the Mythos 5 window being extended.Generated-by: Claude Opus 4.8 (1M context)