Skip to content

feat: Add package command#84

Merged
erezrokah merged 7 commits intomainfrom
feat/docker-publishing
Dec 18, 2023
Merged

feat: Add package command#84
erezrokah merged 7 commits intomainfrom
feat/docker-publishing

Conversation

@maaarcelino
Copy link
Contributor

@maaarcelino maaarcelino commented Nov 21, 2023

This is a proposal for building and packaging Dockerised Python plugins.

The plugin definition would look like this:

Plugin(
    "Test",
    "v0.0.1",
    Options(
        team_name="marcel-team",
        plugin_kind="source",
        dockerfiles=[
            Dockerfile("Dockerfile", [
                BuildTarget("linux", "amd64"),
                BuildTarget("linux", "arm64"),
                BuildTarget("darwin", "amd64"),
                BuildTarget("darwin", "arm64"),
            ]),
            Dockerfile("Dockerfile.windows", [
                BuildTarget("windows", "amd64"),
            ]),
        ]
    )
)

Each Dockerfile has a list of operating systems and architectures it supports. Multiple Dockerfiles can be defined and each is built and saved as a separate tarball.

The package command builds each image and saves it to the dist directory using internal Docker commands. The package.json and docs/overview.md files are also added to the same directory. Everything is then compressed into one plugin.tar.gz file.

erezrokah
erezrokah reviewed Dec 13, 2023
View reviewed changes
erezrokah
erezrokah reviewed Dec 13, 2023
View reviewed changes
erezrokah
erezrokah reviewed Dec 13, 2023
View reviewed changes
@erezrokah erezrokah force-pushed the feat/docker-publishing branch from 2949430 to f632843 Compare December 15, 2023 15:53
erezrokah
erezrokah reviewed Dec 15, 2023
View reviewed changes
cloudquery/sdk/schema/table.py
filtered_tables = []
for t in tt:
filtered_table = copy.deepcopy(t)
for r in filtered_table.relations:
Copy link
Member

@erezrokah erezrokah Dec 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to cloudquery/cloudquery#14637.
This was missing from the Python code and as a result the parent was never set. This means it was never sent via the gRPC call to get the tables.
See relevant Go code in https://github.com/cloudquery/plugin-sdk/blob/36128dd94c73675428b7380c9f7dccb0e6416758/schema/table.go#L201 and https://github.com/cloudquery/plugin-sdk/blob/36128dd94c73675428b7380c9f7dccb0e6416758/schema/table.go#L550

erezrokah
erezrokah reviewed Dec 15, 2023
View reviewed changes
Dockerfile
@@ -0,0 +1,17 @@
FROM python:3.11-slim
Copy link
Member

@erezrokah erezrokah Dec 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a Docker file so we can package the MemDB plugin in the tests

erezrokah
erezrokah reviewed Dec 15, 2023
View reviewed changes
main.py
@@ -0,0 +1,13 @@
import sys
Copy link
Member

@erezrokah erezrokah Dec 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Entry point for the docker file but also makes it easier to do python main.py package -m test "v1.0.0" .

@erezrokah erezrokah marked this pull request as ready for review December 15, 2023 16:03
@erezrokah
Copy link
Member

erezrokah commented Dec 15, 2023

@maaarcelino I believe this is ready now

@erezrokah erezrokah changed the title feat: Docker publishing support proposal feat: Add package command Dec 15, 2023
erezrokah
erezrokah reviewed Dec 15, 2023
View reviewed changes
cloudquery/sdk/plugin/plugin.py
self._opts.dockerfile = "Dockerfile"
if self._opts.build_targets is None:
self._opts.build_targets = [
BuildTarget("linux", "amd64"),
Copy link
Member

@erezrokah erezrokah Dec 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matches our current supported platforms https://github.com/cloudquery/cloudquery/blob/d76f083b5027efd579d1432f59c1d8955182d0a1/.github/workflows/release_plugin.yml#L93

Copy link
Contributor Author

@maaarcelino maaarcelino Dec 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want Darwin targets as defaults? A lot of people do development on Macs (including us).

Copy link
Member

@erezrokah erezrokah Dec 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you can still run Linux images on MacOS. For example Node.js images has only Linux platform:
https://hub.docker.com/_/node/tags?page=1
image

maaarcelino
maaarcelino commented Dec 15, 2023
View reviewed changes
maaarcelino
maaarcelino commented Dec 15, 2023
View reviewed changes
maaarcelino
maaarcelino commented Dec 15, 2023
View reviewed changes
maaarcelino
maaarcelino commented Dec 15, 2023
View reviewed changes
maaarcelino
maaarcelino commented Dec 15, 2023
View reviewed changes
maaarcelino
maaarcelino commented Dec 15, 2023
View reviewed changes
@erezrokah erezrokah force-pushed the feat/docker-publishing branch from d10fbcd to f768597 Compare December 15, 2023 17:30
@erezrokah
Copy link
Member

erezrokah commented Dec 15, 2023

Casing fixed in f768597

@erezrokah erezrokah merged commit 69bd71d into main Dec 18, 2023
@erezrokah erezrokah deleted the feat/docker-publishing branch December 18, 2023 08:37
@cq-bot cq-bot mentioned this pull request Dec 18, 2023
Merged
kodiakhq bot pushed a commit that referenced this pull request Dec 18, 2023
@cq-bot
chore(main): Release v0.1.9 (#87)
d491daf 
🤖 I have created a release *beep* *boop*
---


## [0.1.9](v0.1.8...v0.1.9) (2023-12-18)


### Features

* Add package command ([#84](#84)) ([69bd71d](69bd71d))


### Bug Fixes

* **deps:** Update dependency black to v23.11.0 ([#90](#90)) ([a9bb395](a9bb395))
* **deps:** Update dependency black to v23.12.0 ([#95](#95)) ([1fa9e2c](1fa9e2c))
* **deps:** Update dependency exceptiongroup to v1.2.0 ([#91](#91)) ([2a59001](2a59001))
* **deps:** Update dependency grpcio to v1.59.3 ([#85](#85)) ([4512f59](4512f59))
* **deps:** Update dependency grpcio to v1.60.0 ([#96](#96)) ([4592007](4592007))
* **deps:** Update dependency grpcio-tools to v1.59.3 ([#86](#86)) ([e8eeb7c](e8eeb7c))
* **deps:** Update dependency grpcio-tools to v1.60.0 ([#97](#97)) ([2ffd88b](2ffd88b))
* **deps:** Update dependency numpy to v1.26.2 ([#88](#88)) ([5f8f469](5f8f469))
* **deps:** Update dependency pandas to v2.1.3 ([#89](#89)) ([1a7eaf6](1a7eaf6))
* **deps:** Update dependency pandas to v2.1.4 ([#94](#94)) ([f90efc3](f90efc3))
* **deps:** Update dependency protobuf to v4.25.1 ([#92](#92)) ([cb9105e](cb9105e))
* **deps:** Update dependency structlog to v23.2.0 ([#93](#93)) ([350f2d6](350f2d6))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@erezrokah erezrokah erezrokah left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@maaarcelino @erezrokah