nikitastupin / clairvoyance Goto Github PK

View Code? Open in Web Editor NEW

908.0 11.0 80.0 267 KB

Obtain GraphQL API schema even if the introspection is disabled

License: Apache License 2.0

Python 97.28% JavaScript 1.38% Dockerfile 1.23% Shell 0.12%

graphql security penetration-testing bug-bounty

clairvoyance's People

Stargazers

Watchers

Forkers

slooppe 3esawe bbhunter 0x1bitcrack3r vitthals deepanjalkumar elamaran619 silentsoul04 tjholodnik belane y0k4i-1337 gdraperi njbooher rajat-kanti-dutta gprime31 affilares 47hunt zeroqli touskar cb3d7 hartl3y94 ovelny n0lsecurity darkoneiroi rajivraj cyal1 thiagomayllart ruevaughn blacksper cybernewbies chemouri13 ignitionlab p0intsec chanpu9 arthur4ires iamghostride null-hyp0thesis esenkayshady amber7117 colossalbro polling-repo-continua nullswan mbrg apiliguzov noraj vicnaum dclassified haltcompiler escape-technologies attacker-codeninja joyahamad074 mathdeziel dalaho-bangin kingofthebeat iq-scm cihan-ozcan lithium876 bug-bounty-tester next-generation-graphql-security irsdl oakkaya revan-ar hxlxmjxbbxs 5l1v3r1 filthyshoe ahmetqara 6mile seanpm2001 abdilahrf dancing-goats sandrogarcia heliosrev hritish42 shogun-code itsbrex ev3rpalestine copyleftdev leomatias privat33r-dev

clairvoyance's Issues

Sort fields by name in JSON output

We sort JSON schema by keys:

clairvoyance/clairvoyance/graphql.py

Line 96 in 26eecd1

output = json.dumps(schema, indent=4, sort_keys=True)

but it's not enough for fields to be sorted by name. So we should first sort self.types.keys() and then iterate through them instead of:

clairvoyance/clairvoyance/graphql.py

Line 93 in 26eecd1

for t in self.types.values():

TypeError: init() got an unexpected keyword argument 'allowed_methods'

Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/[email protected]/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/xxx/Code/Python/clairvoyance/clairvoyance/__main__.py", line 96, in <module>
    schema = oracle.clairvoyance(
  File "/Users/xxx/Code/Python/clairvoyance/clairvoyance/oracle.py", line 435, in clairvoyance
    root_typenames = fetch_root_typenames(config)
  File "/Users/xxx/Code/Python/clairvoyance/clairvoyance/oracle.py", line 412, in fetch_root_typenames
    response = graphql.post(
  File "/Users/xxx/Code/Python/clairvoyance/clairvoyance/graphql.py", line 18, in post
    retries = urllib3.util.Retry(
TypeError: __init__() got an unexpected keyword argument 'allowed_methods'

Is this expected?

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

this happening all the time
Also can't import headers, how do I import them correctly?

`cd.yml`: The user 'nikitastupin' isn't allowed to upload to project 'clairvoyance'.

The pypi-push job fails with the following message:

ERROR    HTTPError: 403 Forbidden from https://upload.pypi.org/legacy/          
         The user 'nikitastupin' isn't allowed to upload to project             
         'clairvoyance'. See https://pypi.org/help/#project-name for more       
         information.

It seems that the name "clairvoyance" is already claimed by https://pypi.org/project/clairvoyance/ (https://github.com/ALSchwalm).

We may follow https://pypi.org/help/#project-name-claim and https://peps.python.org/pep-0541/#how-to-request-a-name-transfer to claim the name.

Retry GraphQL query on non-200 status code

Write test case for example below.
Add --retry <num> option with default 3 times and --retry-delay <seconds> with default 5 seconds.
Pass these options via graphql.Config.
Log with INFO level on retry.
❓Unify all GraphQL requests to a single function

Example:

(Pdb) p response.content
b'\n<html><head>\n<meta http-equiv="content-type" content="text/html;charset=utf-8">\n<title>502 Server Error</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Server Error</h1>\n<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n<h2></h2>\n</body></html>\n'
(Pdb) p response.status_code
502

Verify that each word in wordlist conforms to name regex

According to https://spec.graphql.org/June2018/#sec-Names names in GraphQL must match following regex:

[_A-Za-z][_0-9A-Za-z]*

We can skip non-matching names and add flag to disable this verification.

Handling single quotations after multiple_suggestions_re detection

line 38 of the oracle.py should be changed to

valid_fields.add(m.strip('"').strip('\''))

since if a response had multiple field suggestions and their names were in single quotes, the field name would be saved as 'field_name' and not field_name .

Schema validation Not Showing in graphql Voyager

After generating the schema with the tool, i tried on graphql Voyager , but the Voyager not displayed any thing (blank)

Attached video

KeyError: 'Query'

I'm also having the same error when running the tool, however changing the bucket size to 256 didn't solve the issue for me either.
OS: Ubuntu 20.04 LTS
Python: 3.8.10

Here is the output when using the bucket size to 256

python3 -m clairvoyance -o /home/nade/Desktop/schema.json -w google-10000-english-usa.txt https://www.example.com/graphql -vv
[DEBUG][2021-08-21 11:44:25 oracle.py:419]	Root typenames are: {'queryType': None, 'mutationType': None, 'subscriptionType': None}
[DEBUG][2021-08-21 11:44:25 oracle.py:441]	__typename = Query
[DEBUG][2021-08-21 11:44:27 oracle.py:81]	Sent 256 fields, recieved 256 errors in 1.947997 seconds
[DEBUG][2021-08-21 11:44:28 oracle.py:81]	Sent 256 fields, recieved 255 errors in 0.825902 seconds
[DEBUG][2021-08-21 11:44:29 oracle.py:81]	Sent 256 fields, recieved 256 errors in 0.431477 seconds
[DEBUG][2021-08-21 11:44:29 oracle.py:81]	Sent 256 fields, recieved 256 errors in 0.526348 seconds
[DEBUG][2021-08-21 11:44:30 oracle.py:81]	Sent 256 fields, recieved 256 errors in 0.5381 seconds
[DEBUG][2021-08-21 11:44:30 oracle.py:81]	Sent 256 fields, recieved 256 errors in 0.683985 seconds
[DEBUG][2021-08-21 11:44:45 oracle.py:81]	Sent 256 fields, recieved 256 errors in 0.369622 seconds
[DEBUG][2021-08-21 11:44:45 oracle.py:81]	Sent 256 fields, recieved 256 errors in 0.513586 seconds
[DEBUG][2021-08-21 11:44:46 oracle.py:81]	Sent 256 fields, recieved 256 errors in 0.468517 seconds
[DEBUG][2021-08-21 11:44:46 oracle.py:81]	Sent 16 fields, recieved 16 errors in 0.18961 seconds
[DEBUG][2021-08-21 11:44:46 oracle.py:444]	Query.fields = {'_', 'getVariant', 'getCategoryIds', 'getVariants', 'getMemberships', 'getProduct', 'getProductIds', 'getCategory', 'calculateTax', 'version'}
[WARNING][2021-08-21 11:44:46 oracle.py:302]	Unknown error message: '_ disabled'
[DEBUG][2021-08-21 11:44:46 oracle.py:462]	Skip probe_args() for '_' of type 'Boolean'
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/nade/GraphQL pentest/clairvoyance/clairvoyance/__main__.py", line 96, in <module>
    schema = oracle.clairvoyance(
  File "/home/nade/GraphQL pentest/clairvoyance/clairvoyance/oracle.py", line 466, in clairvoyance
    schema.types[typename].fields.append(field)
KeyError: 'Query'

And here is the output when running on default ammount of bucket size (4096)

[DEBUG][2021-08-21 11:46:39 oracle.py:419]	Root typenames are: {'queryType': None, 'mutationType': None, 'subscriptionType': None}
[DEBUG][2021-08-21 11:46:52 oracle.py:441]	__typename = Query
[DEBUG][2021-08-21 11:46:59 oracle.py:81]	Sent 4096 fields, recieved 4095 errors in 6.233555 seconds
[DEBUG][2021-08-21 11:47:06 oracle.py:81]	Sent 4096 fields, recieved 4096 errors in 6.304185 seconds
[DEBUG][2021-08-21 11:47:10 oracle.py:81]	Sent 1808 fields, recieved 1808 errors in 3.456586 seconds
[DEBUG][2021-08-21 11:47:10 oracle.py:444]	Query.fields = {'getVariants', 'getVariant', 'getMemberships', 'calculateTax', 'version', 'getProductIds', 'getCategory', '_', 'getCategoryIds', 'getProduct'}
[WARNING][2021-08-21 11:47:17 oracle.py:194]	Unknown error message: Unknown argument "facilities" on field "Query.getVariants". Did you mean "variantIds"?
[WARNING][2021-08-21 11:47:17 oracle.py:194]	Unknown error message: There can be only one argument named "color".
[WARNING][2021-08-21 11:47:17 oracle.py:194]	Unknown error message: There can be only one argument named "favorite".
[WARNING][2021-08-21 11:47:17 oracle.py:194]	Unknown error message: Unknown argument "variables" on field "Query.getVariants". Did you mean "variantIds"?
[WARNING][2021-08-21 11:47:17 oracle.py:194]	Unknown error message: There can be only one argument named "labor".
[WARNING][2021-08-21 11:47:17 oracle.py:194]	Unknown error message: There can be only one argument named "favorites".
[WARNING][2021-08-21 11:47:26 oracle.py:194]	Unknown error message: Unknown argument "nationwide" on field "Query.getVariants". Did you mean "variantIds"?
[WARNING][2021-08-21 11:47:26 oracle.py:194]	Unknown error message: Unknown argument "variation" on field "Query.getVariants". Did you mean "variantIds"?
[WARNING][2021-08-21 11:47:26 oracle.py:194]	Unknown error message: Unknown argument "variations" on field "Query.getVariants". Did you mean "variantIds"?
[WARNING][2021-08-21 11:47:26 oracle.py:194]	Unknown error message: Unknown argument "validation" on field "Query.getVariants". Did you mean "variantIds"?
[WARNING][2021-08-21 11:47:26 oracle.py:194]	Unknown error message: Unknown argument "warranties" on field "Query.getVariants". Did you mean "variantIds"?
[WARNING][2021-08-21 11:47:26 oracle.py:194]	Unknown error message: There can be only one argument named "harbor".
[WARNING][2021-08-21 11:47:29 oracle.py:194]	Unknown error message: Unknown argument "guarantees" on field "Query.getVariants". Did you mean "variantIds"?
[WARNING][2021-08-21 11:47:29 oracle.py:194]	Unknown error message: Unknown argument "vacancies" on field "Query.getVariants". Did you mean "variantIds"?
[WARNING][2021-08-21 11:47:29 oracle.py:194]	Unknown error message: Unknown argument "variance" on field "Query.getVariants". Did you mean "variantIds"?
[WARNING][2021-08-21 11:47:29 oracle.py:194]	Unknown error message: Unknown argument "varieties" on field "Query.getVariants". Did you mean "variantIds"?
[DEBUG][2021-08-21 11:47:29 oracle.py:452]	Query.getVariants.args = set()
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/nade/GraphQL pentest/clairvoyance/clairvoyance/__main__.py", line 96, in <module>
    schema = oracle.clairvoyance(
  File "/home/nade/GraphQL pentest/clairvoyance/clairvoyance/oracle.py", line 466, in clairvoyance
    schema.types[typename].fields.append(field)
KeyError: 'Query'

Edit: Tried it also in a fresh installed Kali Linux (python 3.9.2) and it also gets the same error

Originally posted by @kleiton0x00 in #16 (comment)

Description

It seems clairvoyance is unable to detect its install dir and load the wordlist from here, instead it tries to load it from the working/current directory.

$ clairvoyance -o /tmp/dvga-schema.json http://noraj.test:5013/graphql
Traceback (most recent call last):
  File "/usr/bin/clairvoyance", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3.10/site-packages/clairvoyance/cli.py", line 109, in cli
    asyncio.run(
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/lib/python3.10/site-packages/clairvoyance/cli.py", line 47, in blind_introspection
    wordlist = wordlist or load_default_wordlist()
  File "/usr/lib/python3.10/site-packages/clairvoyance/cli.py", line 33, in load_default_wordlist
    with open('clairvoyance/wordlist.txt', 'r', encoding='utf-8') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'clairvoyance/wordlist.txt'

So one has to use the -w option even to run the default wordlist.

$ clairvoyance -o /tmp/dvga-schema.json http://noraj.test:5013/graphql -w /usr/lib/python3.10/site-packages/clairvoyance/wordlist.txt 
2022-11-23 15:30:18     INFO    | Starting blind introspection on http://noraj.test:5013/graphql...

Troubleshooting

clairvoyance/clairvoyance/cli.py

Lines 32 to 34 in 137450e

    
           def load_default_wordlist() -> List[str]: 
        
               with open('clairvoyance/wordlist.txt', 'r', encoding='utf-8') as f: 
        
                   return [w.strip() for w in f.readlines() if w.strip()]

I think is could be replaced by something like:

import os
os.path.join(os.path.dirname(__file__), 'wordlist.txt')

I'll PR.

Is the script meant to be very slow?

My first run of it has been going overnight, hasn't produced anything yet.

Console is filled with a bunch of warnings,

[WARNING][2022-03-20 12:42:35 oracle.py:303]	Unknown error message: 'Cannot query field "xxxx" on type "Query".'
[DEBUG][2022-03-20 12:42:35 oracle.py:324]	get_typeref('Cannot query field "xxxx" on type "Query".', 'InputValue') -> None
[WARNING][2022-03-20 12:42:37 oracle.py:303]	Unknown error message: 'Cannot query field "xxxx" on type "Query".'
[DEBUG][2022-03-20 12:42:37 oracle.py:324]	get_typeref('Cannot query field "xxxx" on type "Query".', 'InputValue') -> None
[WARNING][2022-03-20 12:42:39 oracle.py:303]	Unknown error message: 'Cannot query field "xxxx" on type "Query".'
[DEBUG][2022-03-20 12:42:39 oracle.py:324]	get_typeref('Cannot query field "xxxx" on type "Query".', 'InputValue') -> None

Handle 'Field "$s" argument "$s" of type "$s" is required, but it was not provided.' error

While using clairvoyance I noticed those warnings:

[WARNING][2021-04-23 23:58:26 oracle.py:57]     Unknown error message: 'Field "cart" argument "loggedInStatus" of type "LoggedInStatus!" is required, but it was not provided.'
[WARNING][2021-04-23 23:58:26 oracle.py:57]     Unknown error message: 'Field "cart" argument "guid" of type "String!" is required, but it was not provided.'

They can be used to figure arguments

Exception: Unable to get TypeRef for [] in context Field

Hi, this is a very cool project!!

Could you help me figure out what can I do to resolve this? I'm getting an error with command:

python3 -m clairvoyance -o ./schema.json -w wordlist.txt https://www.REDACTED.com/graphql/

Full error:

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/mbrg/install/clairvoyance/clairvoyance/__main__.py", line 4, in <module>
    cli()
  File "/home/mbrg/install/clairvoyance/clairvoyance/cli.py", line 109, in cli
    asyncio.run(
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/mbrg/install/clairvoyance/clairvoyance/cli.py", line 67, in blind_introspection
    schema = await oracle.clairvoyance(
  File "/home/mbrg/install/clairvoyance/clairvoyance/oracle.py", line 503, in clairvoyance
    results = await asyncio.gather(*tasks)
  File "/home/mbrg/install/clairvoyance/clairvoyance/oracle.py", line 438, in explore_field
    typeref = await probe_field_type(
  File "/home/mbrg/install/clairvoyance/clairvoyance/oracle.py", line 358, in probe_field_type
    return await probe_typeref(documents, 'Field')
  File "/home/mbrg/install/clairvoyance/clairvoyance/oracle.py", line 342, in probe_typeref
    raise Exception(e) from e
Exception: Unable to get TypeRef for ['query { ApplicationType }', 'query { ApplicationType { lol } }'] in context Field

Probably related to #20

2022-10-14 20:34:28 INFO | Starting blind introspection on https://site.com/graphql/...
2022-10-14 20:34:29 DEBUG | Root typenames are: {'queryType': None, 'mutationType': None, 'subscriptionType': None}
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/boss/tools/clairvoyance/clairvoyance/main.py", line 4, in
cli()
File "/home/boss/tools/clairvoyance/clairvoyance/cli.py", line 109, in cli
asyncio.run(
File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/boss/tools/clairvoyance/clairvoyance/cli.py", line 67, in blind_introspection
schema = await oracle.clairvoyance(
File "/home/boss/tools/clairvoyance/clairvoyance/oracle.py", line 485, in clairvoyance
typename = await probe_typename(input_document)
File "/home/boss/tools/clairvoyance/clairvoyance/oracle.py", line 402, in probe_typename
raise Exception(f'Expected "{errors}" to match any of "{wrong_field_regexes}".')
Exception: Expected "[{'message': "Validation error of type FieldUndefined: Field 'imwrongfield' in type 'Query' is undefined @ 'imwrongfield'", 'locations': [{'line': 1, 'column': 9}], 'extensions': {'classification': 'ValidationError'}}]" to match any of "['Cannot query field ['"]imwrongfield['"] on type '"['"].', 'Field ['"][_0-9a-zA-Z\[\]!]['"] must not have a selection since type '"['"] has no subfields.', 'Field ['"][_0-9a-zA-Z\[\]!]['"] of type '"['"] must not have a sub selection.']".
2022-10-14 20:34:29 ERROR | Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f744a55f8e0>
2022-10-14 20:34:29 ERROR | Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f744a462e80>, 94397.773572156)]']
connector: <aiohttp.connector.TCPConnector object at 0x7f744a55f670>

command i used python3 -m clairvoyance -vv -o schema.json -w google-10000-english.txt https://site.com/graphql/

SSL Certificate issue

Hello, I have the following error on some hosts with following error:

2023-02-02 02:11:02 WARNING | Error posting to https://host.com/graphql: Cannot connect to host host.com:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'host.com'. (_ssl.c:1131)")]
Could you please add an option to ignore certificates?

Run GitHub Actions only on code and test changes

https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#onpushpull_requestpaths

JSON Requests are not formulating correctly

Most JSON Requests are not formulating correctly when sent. I tried to overcome this using a regex inserting closing brackets and a quote for the incorrect JSON but ultimately it is not consistent and the fix cause errors in other queries. You can see this by sending the tool through a web proxy tool (an added cli option for a proxy would be nice too EDIT: just saw this in a pull request). The tool also errors out very quickly getting rid of all data discovered.

clairvoyance error : connector: <aiohttp.connector.TCPConnector object at 0x7f373f35bdf0>

hello, and thanks for this useful tool, i got an error when runing this tool on https://tdm.dell.com/graphql, please fix the bug (sorry for my english i'm french)

$ clairvoyance https://tdm.dell.com/graphql -c 10 -o graphql.json -H "Apollographql-Client-Name: tdm-ui-mfe"

2023-07-28 13:22:54 INFO | Starting blind introspection on https://tdm.dell.com/graphql...
2023-07-28 13:22:54 INFO | Iteration 1
Traceback (most recent call last):
File "/usr/local/bin/clairvoyance", line 8, in
sys.exit(cli())
File "/usr/local/lib/python3.8/dist-packages/clairvoyance/cli.py", line 142, in cli
asyncio.run(
File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/dist-packages/clairvoyance/cli.py", line 89, in blind_introspection
schema = await oracle.clairvoyance(
File "/usr/local/lib/python3.8/dist-packages/clairvoyance/oracle.py", line 568, in clairvoyance
typename = await probe_typename(input_document)
File "/usr/local/lib/python3.8/dist-packages/clairvoyance/oracle.py", line 487, in probe_typename
return (match.group('typename').replace('[', '').replace(']', '').replace('!', ''))
AttributeError: 'NoneType' object has no attribute 'group'
2023-07-28 13:22:55 ERROR | Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f373f3f1af0>
2023-07-28 13:22:55 ERROR | Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f373dab12e0>, 1663069.501347268)]']
connector: <aiohttp.connector.TCPConnector object at 0x7f373f35bdf0>

`cd.yml`: build Docker images for platforms other than `linux/amd64`

When running on aarch64

sudo docker run --rm nikitastupin/clairvoyance --help

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
exec /bin/sh: exec format error

"Not implemented" exception in TypeRef.to_json()

Input:

(Pdb) pp self.__dict__
{'is_list': True,
 'is_list_item_nullable': True,
 'is_nullable': False,
 'kind': 'SCALAR',
 'list': True,
 'name': 'String',
 'non_null': True,
 'non_null_item': False}

Output:

(Pdb) where
...
> /Users/nikitastupin/Documents/repos/clairvoyance/clairvoyance/graphql.py(234)to_json()
-> raise Exception("Not implemented")

Expected output:

{
  "name": null,
  "kind": "NON_NULL",
  "ofType": {
    "name": null,
    "kind": "LIST",
    "ofType": {
      "name": "String",
      "kind": "SCALAR",
      "ofType": null
    }
  }
}

How apollo-server suggestions works?

Following document returns following suggestions:

mutation { setNameFor }

Cannot query field \"setNameFor\" on type \"Mutation\". Did you mean \"setNameForHome\", \"setNameForCamera\", or \"setNameForHomeSensor\"?

However similar document returns less suggestions:

mutation { setName }

Cannot query field \"setName\" on type \"Mutation\". Did you mean \"setNameForHome\"?

If we would know the algorithm apollo-server uses for suggestion generation we'd potentially speed up guessing of fields, arguments and so on.

TypeError: 'NoneType' object is not subscriptable

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/skroggur/clairvoyance/clairvoyance/__main__.py", line 111, in <module>
    input_document = s.convert_path_to_document(s.get_path_from_root(next))
  File "/home/skroggur/clairvoyance/clairvoyance/graphql.py", line 148, in convert_path_to_document
    elif path[0] == self._schema["mutationType"]["name"]:
TypeError: 'NoneType' object is not subscriptable

What is causing this?

Pacman package does not exist

Pretty self-explanatory. The listed pacman package in README.md does not exist. (at least, it's not there for me)

wrong_field_regexes error

Python version: Python 3.7.5 (Ubuntu)
Also tested on Python 3.8.10 on Windows. Same issue.

python3 -m clairvoyance -w wordlist.txt https://REDACTED/graphql -vv
[DEBUG][2021-09-10 12:11:02 oracle.py:423]      Root typenames are: {'queryType': 'Query', 'mutationType': 'Mutation', 'subscriptionType': None}
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/clairvoyance/clairvoyance/__main__.py", line 97, in <module>
    wordlist, config, input_schema=input_schema, input_document=input_document
  File "/clairvoyance/clairvoyance/oracle.py", line 444, in clairvoyance
    typename = probe_typename(input_document, config)
  File "/clairvoyance/clairvoyance/oracle.py", line 390, in probe_typename
    raise Exception(f"Expected '{errors}' to match any of '{wrong_field_regexes}'.")
Exception: Expected '[{'message': "Cannot query field 'imwrongfield' on type 'Query'.", 'code': 'GRAPHQL_VALIDATION_FAILED'}]' to match any of '['Cannot query field "imwrongfield" on type "(?P<typename>[_0-9a-zA-Z\\[\\]!]*)".', 'Field "[_0-9a-zA-Z\\[\\]!]*" must not have a selection since type "(?P<typename>[_A-Za-z\\[\\]!][_0-9a-zA-Z\\[\\]!]*)" has no subfields.']'.

Probe for Input Object type

Now we're probing only for (1) argument name and (2) it's type. However in case if argument of INPUT_OBJECT type we can probe for (3) fields too.

Impossible to Map Introspection Disabled GQL APIs for These Specific URLS (Apollo Servers Detected)

If anyone could give me a hand -- running into this issue with pretty much every GraphQL enumeration / mapping tool or library for the following URLs. I'm not sure if there is some additional variable I'm missing that I need to be including or potentially a different URL? No clue.

https://api.hypedrop.com/graphql
[Apollo | Introspection Disabled]

https://api.hypeup.com/graphql
[Apollo | Introspection Disabled]

http://api.csgoroll.com/graphql
[Apollo | Introspection Disabled]

Seriously any and all help appreciated as this has been driving me insane!

Thank you so much for an amazing repo & tool. I hope to be able to utilize it!

`cd.yml`: add tags to Docker Hub

To publish 2.0.0, 2.0.1, ... and not just latest https://github.com/docker/build-push-action#inputs

Processing takes way too long

I've been stuck here for 40 minutes. This is a feature request for making the process multithreaded or faster somehow

Exception: Unable to get TypeRef

I issued clairvoyance against an graphql endpoint:

python3 -m clairvoyance -o ./schema.json -w ../../Wordlists/google-10000-english-no-swears.txt https://www.REDACTED.com/graphql/

I'm getting the following error :

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/sim4n6/Desktop/Pentesting-kb/Tools/clairvoyance/clairvoyance/__main__.py", line 96, in <module>
    schema = oracle.clairvoyance(
  File "/home/sim4n6/Desktop/Pentesting-kb/Tools/clairvoyance/clairvoyance/oracle.py", line 454, in clairvoyance
    arg_typeref = probe_arg_typeref(
  File "/home/sim4n6/Desktop/Pentesting-kb/Tools/clairvoyance/clairvoyance/oracle.py", line 353, in probe_arg_typeref
    typeref = probe_typeref(documents, "InputValue", config)
  File "/home/sim4n6/Desktop/Pentesting-kb/Tools/clairvoyance/clairvoyance/oracle.py", line 327, in probe_typeref
    raise Exception(f"Unable to get TypeRef for {documents}")
Exception: Unable to get TypeRef for ['query { lhEvents(filter: 7) }', 'query { lhEvents(filter: {}) }', 'query { lhEvents(filte: 7) }']

Errors with Damn Vulnerable Graphql Application

with master branch:

root@kali:~/Downloads/clairvoyance# python3 -m clairvoyance -w ./google10000.txt http://127.0.0.1:5000/graphql

[WARNING][2021-03-11 22:47:33 oracle.py:57]	Unknown error message: 'Cannot query field "system" on type "Query". Did you mean "pastes", "paste", "systemUpdate" or "systemHealth"?'
[WARNING][2021-03-11 22:47:33 oracle.py:57]	Unknown error message: 'Cannot query field "systems" on type "Query". Did you mean "pastes", "systemUpdate" or "systemHealth"?'
[WARNING][2021-03-11 22:47:33 oracle.py:57]	Unknown error message: 'Field "node" of type "Node" must have a sub selection.'
[WARNING][2021-03-11 22:47:33 oracle.py:57]	Unknown error message: 'Field "node" argument "id" of type "ID!" is required but not provided.'
[WARNING][2021-03-11 22:47:36 oracle.py:57]	Unknown error message: 'Field "paste" of type "PasteObject" must have a sub selection.'
[WARNING][2021-03-11 22:47:38 oracle.py:57]	Unknown error message: 'Cannot query field "systematic" on type "Query". Did you mean "systemUpdate", "systemHealth" or "systemDiagnostics"?'
[WARNING][2021-03-11 22:47:38 oracle.py:57]	Unknown error message: 'Cannot query field "pose" on type "Query". Did you mean "node", "paste" or "pastes"?'
[WARNING][2021-03-11 22:47:38 oracle.py:293]	Unknown error message: 'Field "node" of type "Node" must have a sub selection.'
[WARNING][2021-03-11 22:47:38 oracle.py:293]	Unknown error message: 'Field "node" argument "id" of type "ID!" is required but not provided.'
[WARNING][2021-03-11 22:47:40 oracle.py:188]	Unknown error message: Field "node" of type "Node" must have a sub selection.
[WARNING][2021-03-11 22:47:41 oracle.py:188]	Unknown error message: Field "node" of type "Node" must have a sub selection.
[WARNING][2021-03-11 22:47:41 oracle.py:188]	Unknown error message: Field "node" argument "id" of type "ID!" is required but not provided.
[WARNING][2021-03-11 22:47:41 oracle.py:188]	Unknown error message: Field "node" of type "Node" must have a sub selection.
[WARNING][2021-03-11 22:47:41 oracle.py:188]	Unknown error message: Field "node" argument "id" of type "ID!" is required but not provided.
[WARNING][2021-03-11 22:47:41 oracle.py:293]	Unknown error message: 'Field "node" of type "Node" must have a sub selection.'
[WARNING][2021-03-11 22:47:41 oracle.py:293]	Unknown error message: 'Field "node" of type "Node" must have a sub selection.'
[WARNING][2021-03-11 22:47:41 oracle.py:293]	Unknown error message: 'Argument "id" has invalid value {}.
Expected type "ID", found {}.'
[WARNING][2021-03-11 22:47:41 oracle.py:293]	Unknown error message: 'Field "node" of type "Node" must have a sub selection.'
[WARNING][2021-03-11 22:47:41 oracle.py:293]	Unknown error message: 'Unknown argument "i" on field "node" of type "Query". Did you mean "id"?'
[WARNING][2021-03-11 22:47:41 oracle.py:293]	Unknown error message: 'Field "node" argument "id" of type "ID!" is required but not provided.'
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/Downloads/clairvoyance/clairvoyance/__main__.py", line 89, in <module>
    schema = oracle.clairvoyance(
  File "/root/Downloads/clairvoyance/clairvoyance/oracle.py", line 436, in clairvoyance
    arg_typeref = probe_arg_typeref(
  File "/root/Downloads/clairvoyance/clairvoyance/oracle.py", line 341, in probe_arg_typeref
    typeref = probe_typeref(documents, "InputValue", config)
  File "/root/Downloads/clairvoyance/clairvoyance/oracle.py", line 315, in probe_typeref
    raise Exception(f"Unable to get TypeRef for {documents}")
Exception: Unable to get TypeRef for ['query { node(id: 7) }', 'query { node(id: {}) }', 'query { node(i: 7) }']

Switching to latest Pull request:

root@kali:~/Downloads/clairvoyance# git branch

main
root@kali:~/Downloads/clairvoyance# git branch -a
main
remotes/origin/HEAD -> origin/main
remotes/origin/enhancement-support-input-objects
remotes/origin/fix-issue-9
remotes/origin/fix_non_null_2x
remotes/origin/improvement-retry-on-non-200
remotes/origin/issue-1
remotes/origin/main
remotes/origin/rewrite-system-tests
root@kali:~/Downloads/clairvoyance# git checkout -b enhancement-support-input-objects remotes/origin/enhancement-support-input-objects
Branch 'enhancement-support-input-objects' set up to track remote branch 'enhancement-support-input-objects' from 'origin'.
Switched to a new branch 'enhancement-support-input-objects'
root@kali:~/Downloads/clairvoyance# git branch
enhancement-support-input-objects
main

root@kali:~/Downloads/clairvoyance# python3 -m clairvoyance -w ./google10000.txt http://127.0.0.1:5000/graphql
[WARNING][2021-03-11 22:52:34 oracle.py:57]	Unknown error message: 'Cannot query field "system" on type "Query". Did you mean "pastes", "paste", "systemUpdate" or "systemHealth"?'
[WARNING][2021-03-11 22:52:34 oracle.py:57]	Unknown error message: 'Cannot query field "systems" on type "Query". Did you mean "pastes", "systemUpdate" or "systemHealth"?'
[WARNING][2021-03-11 22:52:34 oracle.py:57]	Unknown error message: 'Field "node" of type "Node" must have a sub selection.'
[WARNING][2021-03-11 22:52:34 oracle.py:57]	Unknown error message: 'Field "node" argument "id" of type "ID!" is required but not provided.'
[WARNING][2021-03-11 22:52:38 oracle.py:57]	Unknown error message: 'Field "paste" of type "PasteObject" must have a sub selection.'
[WARNING][2021-03-11 22:52:39 oracle.py:57]	Unknown error message: 'Cannot query field "systematic" on type "Query". Did you mean "systemUpdate", "systemHealth" or "systemDiagnostics"?'
[WARNING][2021-03-11 22:52:39 oracle.py:57]	Unknown error message: 'Cannot query field "pose" on type "Query". Did you mean "node", "paste" or "pastes"?'
[WARNING][2021-03-11 22:52:39 oracle.py:228]	Unknown error (Field, typeref): Field "pastes" of type "[PasteObject]" must have a sub selection.
[WARNING][2021-03-11 22:52:41 oracle.py:228]	Unknown error (InputValue, name): Field "pastes" of type "[PasteObject]" must have a sub selection.
[WARNING][2021-03-11 22:52:41 oracle.py:228]	Unknown error (InputValue, name): Argument "public" has invalid value 7.
Expected type "Boolean", found 7.
[WARNING][2021-03-11 22:52:43 oracle.py:228]	Unknown error (InputValue, name): Field "pastes" of type "[PasteObject]" must have a sub selection.
[WARNING][2021-03-11 22:52:43 oracle.py:228]	Unknown error (InputValue, name): Field "pastes" of type "[PasteObject]" must have a sub selection.
[WARNING][2021-03-11 22:52:43 oracle.py:228]	Unknown error (InputValue, typeref): Field "pastes" of type "[PasteObject]" must have a sub selection.
[WARNING][2021-03-11 22:52:43 oracle.py:228]	Unknown error (InputValue, typeref): Field "pastes" of type "[PasteObject]" must have a sub selection.
[WARNING][2021-03-11 22:52:43 oracle.py:228]	Unknown error (InputValue, typeref): Argument "public" has invalid value {}.
Expected type "Boolean", found {}.
[WARNING][2021-03-11 22:52:43 oracle.py:228]	Unknown error (InputValue, typeref): Field "pastes" of type "[PasteObject]" must have a sub selection.
[WARNING][2021-03-11 22:52:43 oracle.py:228]	Unknown error (InputValue, typeref): Argument "public" has invalid value 7.
Expected type "Boolean", found 7.
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/Downloads/clairvoyance/clairvoyance/__main__.py", line 91, in <module>
    schema = oracle.clairvoyance(
  File "/root/Downloads/clairvoyance/clairvoyance/oracle.py", line 409, in clairvoyance
    arg_typeref = probe_arg_typeref(
  File "/root/Downloads/clairvoyance/clairvoyance/oracle.py", line 316, in probe_arg_typeref
    typeref = probe_typeref(documents, "InputValue", config)
  File "/root/Downloads/clairvoyance/clairvoyance/oracle.py", line 290, in probe_typeref
    raise Exception(f"Unable to get TypeRef for {documents}")
Exception: Unable to get TypeRef for ['query { pastes(publi: 7) }', 'query { pastes(public: {}) }', 'query { pastes(public: 7) }']

Feature Request : `-proxy`

I would like to request one more option -proxy.

The update would be in here: https://github.com/nikitastupin/clairvoyance/blob/main/clairvoyance/client.py#L46 by adding a proxy support argument as in here https://docs.aiohttp.org/en/stable/client_advanced.html?highlight=proxy#proxy-support

Thank you in advance.

Validate GraphQL JSON Schema

I've got a Schema resulting from clairvoyance. It is validated using JSON.parse().
However, don't know what it is not working on GraphQL-voyager or Postman.
The latter returns the following error Invalid Schema supplied: The provided input schema is syntactically invalid when TestSuite is performed for graphQL JSON schema.

Do you have any idea on a way to validate the GRAPHQL JSON schema ?

By the way, a little tool named graphql-path-enum is capable of finding a lot of paths with ease.

Determine bucket_size dynamically

It's fixed to 2048 fields but we can do better!

client_session: <aiohttp.client.ClientSession object at 0x7f3307f050d0>

hello, when i run clairvoyance on some targets like this (https://ctm-cssit-ps3-dmz.us.dell.com/) i got and error, how can i fix it ?
error :

root@aliwjpi:# clairvoyance "https://ctm-cssit-ps3-dmz.us.dell.com/graphql" -o schema.json -c 1
2023-04-21 05:12:24 INFO | Starting blind introspection on https://ctm-cssit-ps3-dmz.us.dell.com/graphql...
2023-04-21 05:12:24 INFO | Iteration 1
Traceback (most recent call last):
File "/usr/local/bin/clairvoyance", line 8, in
sys.exit(cli())
File "/usr/local/lib/python3.8/dist-packages/clairvoyance/cli.py", line 142, in cli
asyncio.run(
File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/dist-packages/clairvoyance/cli.py", line 89, in blind_introspection
schema = await oracle.clairvoyance(
File "/usr/local/lib/python3.8/dist-packages/clairvoyance/oracle.py", line 568, in clairvoyance
typename = await probe_typename(input_document)
File "/usr/local/lib/python3.8/dist-packages/clairvoyance/oracle.py", line 487, in probe_typename
return (match.group('typename').replace('[', '').replace(']', '').replace('!', ''))
AttributeError: 'NoneType' object has no attribute 'group'
2023-04-21 05:12:25 ERROR | Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f651f7840d0>

Add SECURITY.txt

Vulnerabilities can be reported via https://github.com/nikitastupin/clairvoyance/security/advisories

deleted

simplejson.errors.JSONDecodeError

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "G:\Python\clairvoyance-main\clairvoyance\__main__.py", line 96, in <module>
    schema = oracle.clairvoyance(
  File "G:\Python\clairvoyance-main\clairvoyance\oracle.py", line 447, in clairvoyance
    valid_mutation_fields = probe_valid_fields(wordlist, config, input_document)
  File "G:\Python\clairvoyance-main\clairvoyance\oracle.py", line 80, in probe_valid_fields
    errors = response.json()["errors"]
  File "C:\Users\Folder\AppData\Roaming\Python\Python38\site-packages\requests\models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\simplejson\__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "C:\ProgramData\Anaconda3\lib\site-packages\simplejson\decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "C:\ProgramData\Anaconda3\lib\site-packages\simplejson\decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Executing it for GET requests only

Hi, how to run the tool for the cases where endpoint supports GET requests only.

`.github/workflows/*`: set minimal required permissions

https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs

Idea: add brute-force as fallback

Sometimes suggestions aren't working but it's nevertheless possible to retrieve a portion of the schema. This can be achieved by the good old brute-force technique. To amplify its efficiency permutations may be used as with subdomain enumeration techniques.

We may also probe if suggestions are enabled and if they aren't fallback to brute-force.

Though it's good to have a flag to choose the method manually.

The endpoint requires authorization keys

What should I do if my GraphQL endpoint requires cookies and authorization keys, how can I add this for analysis?

Infinite Loop When TypeRef References Parent TypeRef

Clairvoyance cannot handle when a schema typeref back-references a typeref. The tool ends up stuck in an infinite loop and never progresses.

The tool gets caught in get_path_from_root() in graphql.py where there is a while loop that never completes. I ended up monkey-patching the oracle.py to avoid specific TypeName's that have this back reference.

Attached is a screenshot of a schema which does this. I had to kill the tool and the resulting schema showed the issue when thrown into a visualizer.

Utilise obtained names for probing

For example, we can break maxAtmospheringSpeed to max, maxAtmosphering, AtmospheringSpeed, max, Atmosphering and Speed names and use them for probing!

We can also add these to wordlist so they will be used in consequent probes 😃

Unable to use headers with docker container

Great tool,

While using the local installation of clairvoyance via pip, it works perfectly with headers.

However, with the docker container there's a bug.

Removing the header flag solves it but sometimes graphql endpoints require authentication.

Errors running tests

Hi! Awesome project 🙏

Was trying to do python3 -m unittest tests/*_test.py using main branch and am getting:

======================================================================
ERROR: graphql_test (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: graphql_test
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
ModuleNotFoundError: No module named 'tests.graphql_test'


======================================================================
ERROR: oracle_test (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: oracle_test
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
ModuleNotFoundError: No module named 'tests.oracle_test'


----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (errors=2)

unittest package is included by default so no problem there:

fisher@Fishers-MBP clairvoyance % python -m unittest

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

On Python 3.9.4

Any idea?

Also since I haven't been able to actually play around with the tool, can we use the generated file (-o) as input in subsequent runs?

Thanks!

What is this error ` WARNING | Error posting to https://domain.tld/graphql: Expecting value: line 1 column 1 (char 0)`

Fingerprint the server before probing

Probably, we can use https://github.com/Escape-Technologies/graphdna.

Warn a user if the server is not supported (because Clairvoyance is not guaranteed to work with every server).

Start with introspection query

It's assumed that user has already checked that case, but it might be useful to check with introspection query prior to the main part of trying to bruteforce.

For the flexibility's sake and due to potential sensitivity of blue team solutions, this part should be optional. But since fast profile is default, it can be enabled by default as well.

Why introspection query use ofType in TypeRef more than 4 times?

fragment TypeRef on __Type {
kind
name
ofType {
  kind
  name
  ofType {
    kind
    name
    ofType {
      kind
      name
      ofType {
        kind
        name
        ofType {
          kind
          name
          ofType {
            kind
            name
            ofType {
              kind
              name
            }
          }
        }
      }
    }
  }
}

Unknown error message

I was getting this kind of error:
DEBUG | Unknown error message for valid_args: 'Unknown argument "forum" on field "Query.developers". Did you mean "first" or "sort"?'

Maybe worth to add something like "Did you mean" to regex and extract fields from there if there are quotes?

	def load_default_wordlist() -> List[str]:
	with open('clairvoyance/wordlist.txt', 'r', encoding='utf-8') as f:
	return [w.strip() for w in f.readlines() if w.strip()]