pydataverse upgrade broke `OnlineDataverseDataset.upload_file()`

[mih]

We pass replace_file() the following instructions

       datafile.set({
            # if we do not give `label`, it would use the local filename
            'label': remote_path.name,
            # the model enforces this property, despite `label` being the
            # effective setter, and despite it being ignore and replaced
            # we the local filename
            'filename': remote_path.name,
            'directoryLabel': str(remote_path.parent),
            'pid': self._dsid,
        })

It appears that directoryLabel gets ignore with pyDataverse==0.3.3

[mih]

What might cause the problem here is a "fix" made for gdcc/pyDataverse#171 that came with v0.3.2 in gdcc/pyDataverse#174

I am not sure what is happening here. I thought the API of replace_file changed, but the documentation of identifier is misleading

It is not the identifier of the dataset, but of the file.

[mih]

I cannot determine what pid in the metadata record is supposed to be (it is mandatory). I tried switching it to be the file id, but that has no impact.

[mih]

And something is out-of-sync with the underlying dataverse API. Here is the validation error I get when intentionally butchering the metadata record:

E           jsonschema.exceptions.ValidationError: 'filename' is a required property
E           
E           Failed validating 'required' in schema:
E               {'$schema': 'http://json-schema.org/draft-07/schema',
E                '$id': 'https://github.com/GDCC/pyDataverse/schemas/json/datafile_upload_schema.json',
E                'type': 'object',
E                'title': 'Datafile JSON upload schema',
E                'description': 'Describes the full Datafile metadata JSON file '
E                               'structure for a Dataverse API upload.',
E                'default': {},
E                'required': ['pid', 'filename'],
E                'additionalProperties': False,
E                'properties': {'description': {'$id': '#/properties/description',
E                                               'type': 'string'},
E                               'categories': {'$id': '#/properties/categories',
E                                              'type': 'array',
E                                              'additionalItems': False,
E                                              'items': {'anyOf': [{'$id': '#/properties/categories/items/anyOf/0',
E                                                                   'type': 'string'}],
E                                                        '$id': '#/properties/categories/items'}},
E                               'restrict': {'$id': '#/properties/restrict',
E                                            'type': 'boolean'},
E                               'pid': {'$id': '#/properties/pid', 'type': 'string'},
E                               'filename': {'$id': '#/properties/filename',
E                                            'type': 'string'},
E                               'label': {'$id': '#/properties/label',
E                                         'type': 'string'},
E                               'directoryLabel': {'$id': '#/properties/directoryLabel',
E                                                  'type': 'string'}}}
E           

However https://guides.dataverse.org/en/latest/api/native-api.html#replacing-files shows that a metadata record like this is possible:

{"description":"My description.","categories":["Data"],"forceReplace":false}

It has neither of the properties that the pydataverse schema requires, and a forceReplace that it does not allow.

The tests added for replace_datafile() in pyDataverse in gdcc/pyDataverse@7242082 bypass this issue by using a plain json.dumps()

[JR-1991]

Indeed, the documentation is just confusing, because it says "... of the dataset" when it is actually of the file.

Regarding the directoryLabel issue, I am a bit puzzled, because the body of the request is not touched at any point. Do other properties of the body sent change? E.g. the description?

Trying to reproduce locally now and will get back soon.

[JR-1991]

I found the bug! Dataverse requires the metadata to be sent as form-data. It seems that using the json argument in HTTPX is not enough. However, when passed to the data parameter, the issue is resolved, and all metadata is updated correctly.

I will verify if this change does not cause any problems in other areas and will then prepare a pull request to fix this as soon as possible. 😊

[JR-1991]

I've created a draft pull request. Can you verify that it is working correctly now?

gdcc/pyDataverse#203

[mih]

With the tentative fix in pydataverse we arrive at 5 remaining failures:

FAILED datalad_dataverse/tests/test_add_sibling_dataverse.py::test_asdv_invalid_calls - assert 'doi:no-ffing-datalad-way-this-exists not found' in "CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false annex initre...
FAILED datalad_dataverse/tests/test_pydataverse.py::test_file_handling - httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://demo.dataverse.org/api/v1/datasets/:persistentId/add?persistentId=doi%3A10...
FAILED datalad_dataverse/tests/test_pydataverse.py::test_file_removal - AssertionError: failed to upload file 400: {'status': 'ERROR', 'message': 'Error in parsing provided json'}
FAILED datalad_dataverse/tests/test_remote.py::test_remote[yes] - datalad.runner.exception.CommandError: CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false annex testremote --fast mydv -c a...
FAILED datalad_dataverse/tests/test_remote.py::test_remote[no] - datalad.runner.exception.CommandError: CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false annex testremote --fast mydv -c a...

It looks a bit as if the next blocker may be in the present implementation of file removal. Needs more investigation.

[JR-1991]

Can you provide details of the failed tests? I will be on vacation from today until over next week Monday. Happy to get these fixed asap when back 😊

[mih]

Thank you. I will, once I know and cannot figure it out myself. Happy vacation!

[adswa]

I plan to look at the failing tests this afternoon

[adswa]

I'll comment here for now, but I'm happy to re-file anything of relevance in more appropriate places.

I found an issue with pydataverse.api.upload_datafile, or rather a mismatch between actual behavior and behavior expected according to the Add a file to a dataset API. In datalad-dataverse's test_file_removal (and test_file_handling) , we call upload_file using only the dataset identifier and filename

datalad-dataverse/datalad_dataverse/tests/test_pydataverse.py

Lines 161 to 164 in 0621e2d

	response = dataverse_admin_api.upload_datafile(
	identifier=dataverse_dataset,
	filename=fpath,
	)

The signature of pydataverse.api.upload_datafile and the API guide - at least to me - read as if this is all that's required. However, I found that pydataverse.api.upload_datafile's json_str can not be None, otherwise the API returns a 400 status code. Debugging inside of pydataverse.api.upload_datafile: The call fails when json_str is None (its default value), but succeeds when its the example payload from the Python script in the API guide, or even any key-value:

(Pdb) p url
'https://demo.dataverse.org/api/v1/datasets/:persistentId/add?persistentId=doi:10.70122/FK2/SG3AT3'
(Pdb) n
> /home/adina/repos/pyDataverse/pyDataverse/api.py(1851)upload_datafile()
-> return self.post_request(
(Pdb) p files
{'file': <_io.BufferedReader name='/tmp/pytest-of-adina/pytest-3/test_file_removal0/dummy.txt'>}
(Pdb) p json_str
None
(Pdb) self.post_request(url, data={"jsonData": json_str}, files=files, auth=True)
<Response [400 Bad Request]>
(Pdb) self.post_request(url, data={"jsonData": '{"description": "Blue skies!", "categories": ["Lily", "Rosemary", "Jack of Hearts"]}'}, files=files, auth=True)
<Response [200 OK]>
(Pdb) self.post_request(url, data={"jsonData": '{"some":"thing"}'}, files=files, auth=True)
<Response [200 OK]>

(I was also a bit confused about the 200 response as the API Guide says "You should expect a 201 (“CREATED”) response and JSON indicating the database id that has been assigned to your newly uploaded file.", but datalad-dataverse code was already adjusted to this and checks for a 200).

Simply adding a random JSON payload to the upload_datafile call fixes two of the failing tests (test_file_handling, test_file_removal)

diff --git a/datalad_dataverse/tests/test_pydataverse.py b/datalad_dataverse/te>
index 93e1719..a6a94fd 100644
--- a/datalad_dataverse/tests/test_pydataverse.py
+++ b/datalad_dataverse/tests/test_pydataverse.py
@@ -83,14 +83,16 @@ def check_duplicate_file_deposition(api, dsid, tmp_path):
 
     response = api.upload_datafile(
         identifier=dsid,
-        filename=tmp_path / 'nonunique1.txt'
+        filename=tmp_path / 'nonunique1.txt',
+        json_str='{"some":"thing"}'
     )
     # we do not expect issues here
     response.raise_for_status()
     # now upload the second file with the same content
     response = api.upload_datafile(
         identifier=dsid,
-        filename=tmp_path / 'nonunique2.txt'
+        filename=tmp_path / 'nonunique2.txt',
+        json_str='{"some":"thing"}'
     )
     response.raise_for_status()
 
@@ -109,6 +111,7 @@ def check_upload(api, dsid, fcontent, fpath, src_md5):
     response = api.upload_datafile(
         identifier=dsid,
         filename=fpath,
+        json_str='{"some":"thing"}'
     )
     # worked
     assert response.status_code == 200
@@ -161,6 +164,7 @@ def test_file_removal(
     response = dataverse_admin_api.upload_datafile(
         identifier=dataverse_dataset,
         filename=fpath,
+        json_str='{"some":"thing"}'
     )
     # worked
     assert response.status_code == 200, \
@@ -184,6 +188,7 @@ def test_file_removal(
     response = dataverse_admin_api.upload_datafile(
         identifier=dataverse_dataset,
         filename=fpath,
+        json_str='{"some":"thing"}'
     )
     assert response.status_code == 200, \
         f"failed to upload file {response.status_code}: {response.json()}"