paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 42 Packages Projects Releases Wiki Activity

Merge pull request #4232 from backstage/rugvip/conf

Browse Source 
config-loader: refactor transform logic and add support for env var substitutions
This commit is contained in:
Patrik Oldsberg
2021-01-25 12:11:43 +01:00
committed by GitHub
parent 5a976ce03c 26f41ef21d
commit 5eb4bdf038
26 changed files with 703 additions and 730 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.changeset/five-games-grin.md
+5
View File
@@ -0,0 +1,5 @@
---
'@backstage/cli': patch
---
Add `--lax` option to `config:print` and `config:check`, which causes all environment variables to be assumed to be set.
.changeset/happy-crabs-punch.md
+12
View File
@@ -0,0 +1,12 @@
---
'@backstage/config-loader': patch
---
Added support for environment variable substitutions in string configuration values using a `${VAR}` placeholder. All environment variables must be available, or the entire expression will be evaluated to `undefined`. To escape a substitution, use `$${...}`, which will end up as `${...}`.
For example:
```yaml
app:
baseUrl: https://${BASE_HOST}
```
.changeset/lucky-guests-mate.md
+5
View File
@@ -0,0 +1,5 @@
---
'@backstage/config-loader': minor
---
Removed support for the deprecated `$data` placeholder.
.changeset/tender-parrots-itch.md
+5
View File
@@ -0,0 +1,5 @@
---
'@backstage/config-loader': minor
---
Enable further processing of configuration files included using the `$include` placeholder. Meaning that for example for example `$env` includes will be processed as usual in included files.
docs/conf/index.md
+2 -2
View File
@@ -18,8 +18,8 @@ allowing for customization.
Configuration is stored in YAML files where the defaults are `app-config.yaml`
and `app-config.local.yaml` for local overrides. Other sets of files can by
loaded by passing `--config <path>` flags. The configuration files themselves
contain plain YAML, but with support for loading in secrets from various sources
using for example `$env` and `$file` keys.
contain plain YAML, but with support for loading in data and secrets from
various sources using for example `$env` and `$file` keys.
It is also possible to supply configuration through environment variables, for
example `APP_CONFIG_app_baseUrl=https://staging.example.com`. However these
docs/conf/writing.md
+37 -23
View File
@@ -97,13 +97,13 @@ order:
- If no config flags are provided, `app-config.local.yaml` has higher priority
than `app-config.yaml`.
## Secrets and Dynamic Data
## Includes and Dynamic Data
Secrets are supported via special data loading keys that are prefixed with `$`,
which in turn provide a number of different ways to read in secrets. To load a
configuration value as a secret, supply an object with one of the special secret
keys, for example `$env` or `$file`. A full list of supported secret keys can be
found below. For example, the following will read the config key
Includes are supported via special data loading keys that are prefixed with `$`,
which in turn provide a number of different ways to read in data. To load in an
external configuration value, supply an object with one of the special include
keys, for example `$env` or `$file`. A full list of supported include keys can
be found below. For example, the following will read the config key
`backend.mySecretKey` from the environment variable `MY_SECRET_KEY`:
```yaml
@@ -114,28 +114,26 @@ backend:
With the above configuration, calling `config.getString('backend.mySecretKey')`
will return the value of the environment variable `MY_SECRET_KEY` when the
backend started up. All secrets are loaded at startup, so changing the contents
of secret files or environment variables will not be reflected at runtime.
backend started up. All includes are loaded at startup, so changing the contents
of files or environment variables will not be reflected at runtime.
As hinted at, secrets can be loaded from a bunch of different sources, and can
be extended with more. Below is a list of the currently supported methods for
loading secrets.
Below is a list of the currently supported methods for loading includes.
### Env Secrets
### Env Includes
This reads a secret from an environment variable. For example, the following
config loads the secret from the `MY_SECRET` env var.
This reads a string value from an environment variable. For example, the
following configuration loads the string value from the `MY_SECRET` environment
variable.
```yaml
$env: MY_SECRET
```
### File Secrets
### File Includes
This reads a secret from the entire contents of a file. The file path is
relative to the `app-config.yaml` the defines the secrets. For example, the
following reads the contents of `my-secret.txt` relative to the config file
itself:
This reads a string value from the entire contents of a text file. The file path
is relative to the source config file. For example, the following reads the
contents of `my-secret.txt` relative to the config file itself:
```yaml
$file: ./my-secret.txt
@@ -143,10 +141,10 @@ $file: ./my-secret.txt
### Including Files
The `$include` keyword can be used to load in JSON data from an external file.
It's able to load and parse data from `.json`, `.yml`, and `.yaml` files. It's
also possible to include a url fragment (`#`) to point to a value at the given
path in the file.
The `$include` keyword can be used to load configuration values from an external
file. It's able to load and parse data from `.json`, `.yml`, and `.yaml` files.
It's also possible to include a url fragment (`#`) to point to a value at the
given path in the file, using a dot-separated list of keys.
For example, the following would read `my-secret-key` from `my-secrets.json`:
@@ -163,3 +161,19 @@ Example `my-secrets.json` file:
}
}
```
## Environment Variable Substitution
Configuration files support environment variable substitution via a `${MY_VAR}`
syntax. For example:
```yaml
app:
baseUrl: https://${HOST}
```
Note that all environment variables must be available, or the entire
configuration value will evaluate to `undefined`.
The substitution syntax can be escaped using `$${...}`, which will be resolved
as `${...}`.
packages/cli/src/commands/config/print.ts
+1
View File
@@ -24,6 +24,7 @@ export default async (cmd: Command) => {
const { schema, appConfigs } = await loadCliConfig({
args: cmd.config,
fromPackage: cmd.package,
mockEnv: cmd.lax,
});
const visibility = getVisibilityOption(cmd);
const data = serializeConfigData(appConfigs, schema, visibility);
packages/cli/src/commands/config/validate.ts
+1
View File
@@ -21,5 +21,6 @@ export default async (cmd: Command) => {
await loadCliConfig({
args: cmd.config,
fromPackage: cmd.package,
mockEnv: cmd.lax,
});
};
packages/cli/src/commands/index.ts
+2
View File
@@ -153,6 +153,7 @@ export function registerCommands(program: CommanderStatic) {
'--package <name>',
'Only load config schema that applies to the given package',
)
.option('--lax', 'Do not require environment variables to be set')
.option('--frontend', 'Print only the frontend configuration')
.option('--with-secrets', 'Include secrets in the printed configuration')
.option(
@@ -169,6 +170,7 @@ export function registerCommands(program: CommanderStatic) {
'--package <name>',
'Only load config schema that applies to the given package',
)
.option('--lax', 'Do not require environment variables to be set')
.option(...configOption)
.description(
'Validate that the given configuration loads and matches schema',
packages/cli/src/lib/config.ts
+4 -1
View File
@@ -21,6 +21,7 @@ import { paths } from './paths';
type Options = {
args: string[];
fromPackage?: string;
mockEnv?: boolean;
};
export async function loadCliConfig(options: Options) {
@@ -40,7 +41,9 @@ export async function loadCliConfig(options: Options) {
});
const appConfigs = await loadConfig({
env: process.env.APP_ENV ?? process.env.NODE_ENV ?? 'production',
experimentalEnvFunc: options.mockEnv
? async name => process.env[name] || 'x'
: undefined,
configRoot: paths.targetRoot,
configPaths,
});
packages/config-loader/src/lib/index.ts
+1 -2
View File
@@ -14,7 +14,6 @@
* limitations under the License.
*/
export { readConfigFile } from './reader';
export { readEnvConfig } from './env';
export { readSecret } from './secrets';
export * from './transform';
export * from './schema';
packages/config-loader/src/lib/reader.test.ts
-181
View File
@@ -1,181 +0,0 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { readConfigFile } from './reader';
import { ReaderContext, ReadSecretFunc } from './types';
function memoryFiles(files: { [path: string]: string }) {
return async (path: string) => {
if (path in files) {
return files[path];
}
throw new Error(`File not found, ${path}`);
};
}
const mockContext: ReaderContext = {
env: {},
readFile: jest.fn(),
readSecret: jest.fn(),
};
describe('readConfigFile', () => {
it('should read a plain config file', async () => {
const readFile = memoryFiles({
'./app-config.yaml':
'app: { title: "Test", x: 1, y: [null, true], z: null }',
});
const config = readConfigFile('./app-config.yaml', {
...mockContext,
readFile,
});
await expect(config).resolves.toEqual({
data: {
app: {
title: 'Test',
x: 1,
y: [true],
},
},
context: 'app-config.yaml',
});
});
it('should error out if the config file has invalid syntax', async () => {
const readFile = memoryFiles({
'./app-config.yaml': 'app: { title: ]',
});
const config = readConfigFile('./app-config.yaml', {
...mockContext,
readFile,
});
await expect(config).rejects.toThrow('Flow map contains an unexpected ]');
});
it('should error out if config is not an object', async () => {
const readFile = memoryFiles({
'./app-config.yaml': '[]',
});
const config = readConfigFile('./app-config.yaml', {
...mockContext,
readFile,
});
await expect(config).rejects.toThrow('Expected object at config root');
});
it('should read secrets', async () => {
const readFile = memoryFiles({
'./app-config.yaml': 'app: { $file: "./my-secret" }',
});
const readSecret = jest.fn().mockResolvedValue('secret');
const config = readConfigFile('./app-config.yaml', {
...mockContext,
readFile,
readSecret: readSecret as ReadSecretFunc,
});
await expect(config).resolves.toEqual({
data: {
app: 'secret',
},
context: 'app-config.yaml',
});
expect(readSecret).toHaveBeenCalledWith('.app', {
file: './my-secret',
});
});
it('should not allow keys adjacent to secrets', async () => {
const readFile = memoryFiles({
'./app-config.yaml': 'app: { extraKey: 3, $file: "./my-secret" }',
});
const readSecret = jest.fn().mockResolvedValue('secret');
const config = readConfigFile('./app-config.yaml', {
...mockContext,
readFile,
readSecret: readSecret as ReadSecretFunc,
});
await expect(config).rejects.toThrow(
"Secret key '$file' has adjacent keys at .app",
);
expect(readSecret).not.toHaveBeenCalled();
});
it('should read deprecated secrets', async () => {
const readFile = memoryFiles({
'./app-config.yaml': 'app: { $secret: { file: "./my-secret" } }',
});
const readSecret = jest.fn().mockResolvedValue('secret');
const config = readConfigFile('./app-config.yaml', {
...mockContext,
readFile,
readSecret: readSecret as ReadSecretFunc,
});
await expect(config).resolves.toEqual({
data: {
app: 'secret',
},
context: 'app-config.yaml',
});
expect(readSecret).toHaveBeenCalledWith('.app', {
file: './my-secret',
});
});
it('should require deprecated secrets to be objects', async () => {
const readFile = memoryFiles({
'./app-config.yaml': 'app: { $secret: ["wrong-type"] }',
});
const readSecret = jest.fn().mockResolvedValue('secret');
const config = readConfigFile('./app-config.yaml', {
...mockContext,
readFile,
readSecret: readSecret as ReadSecretFunc,
});
expect(readSecret).not.toHaveBeenCalled();
await expect(config).rejects.toThrow(
'Expected object at secret .app.$secret',
);
});
it('should forward secret reading errors', async () => {
const readFile = memoryFiles({
'./app-config.yaml': 'app: { $secret: {} }',
});
const readSecret = jest.fn().mockRejectedValue(new Error('NOPE'));
const config = readConfigFile('./app-config.yaml', {
...mockContext,
readFile,
readSecret: readSecret as ReadSecretFunc,
});
await expect(config).rejects.toThrow('Invalid secret at .app: NOPE');
});
});
packages/config-loader/src/lib/reader.ts
-111
View File
@@ -1,111 +0,0 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { AppConfig, JsonObject, JsonValue } from '@backstage/config';
import { basename } from 'path';
import yaml from 'yaml';
import { ReaderContext } from './types';
import { isObject } from './utils';
/**
* Reads and parses, and validates, and transforms a single config file.
* The transformation rewrites any special values, like the $secret key.
*/
export async function readConfigFile(
filePath: string,
ctx: ReaderContext,
): Promise<AppConfig> {
const configYaml = await ctx.readFile(filePath);
const config = yaml.parse(configYaml);
const context = basename(filePath);
async function transform(
obj: JsonValue,
path: string,
): Promise<JsonValue | undefined> {
if (typeof obj !== 'object') {
return obj;
} else if (obj === null) {
return undefined;
} else if (Array.isArray(obj)) {
const arr = new Array<JsonValue>();
for (const [index, value] of obj.entries()) {
const out = await transform(value, `${path}[${index}]`);
if (out !== undefined) {
arr.push(out);
}
}
return arr;
}
// TODO(Rugvip): This form of declaring secrets is deprecated, warn and remove in the future
if ('$secret' in obj) {
console.warn(
`Deprecated secret declaration at '${path}' in '${context}', use $env, $file, etc. instead`,
);
if (!isObject(obj.$secret)) {
throw TypeError(`Expected object at secret ${path}.$secret`);
}
try {
return await ctx.readSecret(path, obj.$secret);
} catch (error) {
throw new Error(`Invalid secret at ${path}: ${error.message}`);
}
}
// Check if there's any key that starts with a '$', in that case we treat
// this entire object as a secret.
const [secretKey] = Object.keys(obj).filter(key => key.startsWith('$'));
if (secretKey) {
if (Object.keys(obj).length !== 1) {
throw new Error(
`Secret key '${secretKey}' has adjacent keys at ${path}`,
);
}
try {
return await ctx.readSecret(path, {
[secretKey.slice(1)]: obj[secretKey],
});
} catch (error) {
throw new Error(`Invalid secret at ${path}: ${error.message}`);
}
}
const out: JsonObject = {};
for (const [key, value] of Object.entries(obj)) {
// undefined covers optional fields
if (value !== undefined) {
const result = await transform(value, `${path}.${key}`);
if (result !== undefined) {
out[key] = result;
}
}
}
return out;
}
const finalConfig = await transform(config, '');
if (!isObject(finalConfig)) {
throw new TypeError('Expected object at config root');
}
return { data: finalConfig, context };
}
packages/config-loader/src/lib/secrets.test.ts
-162
View File
@@ -1,162 +0,0 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { readSecret } from './secrets';
import { ReaderContext } from './types';
const ctx: ReaderContext = {
env: {
SECRET: 'my-secret',
},
readSecret: jest.fn(),
async readFile(path) {
const content = ({
'my-secret': 'secret',
'my-data.json': '{"a":{"b":{"c":42}}}',
'my-data.yaml': 'some:\n yaml:\n key: 7',
'my-data.yml': 'different: { key: hello }',
'invalid.yaml': 'foo: [}',
} as { [key: string]: string })[path];
if (!content) {
throw new Error('File not found!');
}
return content;
},
};
describe('readSecret', () => {
it('should read file secrets', async () => {
await expect(readSecret({ file: 'my-secret' }, ctx)).resolves.toBe(
'secret',
);
await expect(readSecret({ file: 'no-secret' }, ctx)).rejects.toThrow(
'File not found!',
);
});
it('should read present env secrets', async () => {
await expect(readSecret({ env: 'SECRET' }, ctx)).resolves.toBe('my-secret');
await expect(readSecret({ env: 'NO_SECRET' }, ctx)).resolves.toBe(
undefined,
);
});
it('should read data secrets', async () => {
// Deprecated object form
await expect(
readSecret({ data: 'my-data.json', path: 'a.b.c' }, ctx),
).resolves.toBe('42');
await expect(
readSecret({ data: 'my-data.yaml', path: 'some.yaml.key' }, ctx),
).resolves.toBe('7');
await expect(
readSecret({ data: 'my-data.yml', path: 'different.key' }, ctx),
).resolves.toBe('hello');
await expect(
readSecret({ data: 'no-data.yml', path: 'different.key' }, ctx),
).rejects.toThrow('File not found!');
// New format with path in fragment
await expect(readSecret({ data: 'my-data.json#a.b.c' }, ctx)).resolves.toBe(
'42',
);
await expect(
readSecret({ data: 'my-data.yaml#some.yaml.key' }, ctx),
).resolves.toBe('7');
await expect(
readSecret({ data: 'my-data.yml#different.key' }, ctx),
).resolves.toBe('hello');
await expect(
readSecret({ data: 'no-data.yml#different.key' }, ctx),
).rejects.toThrow('File not found!');
});
it('should include extra files', async () => {
// New format with path in fragment
await expect(
readSecret({ include: 'my-data.json#a.b.c' }, ctx),
).resolves.toBe(42);
await expect(
readSecret({ include: 'my-data.json#a.b' }, ctx),
).resolves.toEqual({ c: 42 });
await expect(
readSecret({ include: 'my-data.yaml#some.yaml.key' }, ctx),
).resolves.toBe(7);
await expect(readSecret({ include: 'my-data.yaml' }, ctx)).resolves.toEqual(
{
some: { yaml: { key: 7 } },
},
);
await expect(
readSecret({ include: 'my-data.yaml#' }, ctx),
).resolves.toEqual({
some: { yaml: { key: 7 } },
});
await expect(
readSecret({ include: 'my-data.yml#different.key' }, ctx),
).resolves.toBe('hello');
await expect(
readSecret({ include: 'no-data.yml#different.key' }, ctx),
).rejects.toThrow('File not found!');
await expect(
readSecret({ include: 'my-data.yml#missing.key' }, ctx),
).rejects.toThrow('Value is not an object at missing in my-data.yml');
await expect(readSecret({ include: 'invalid.yaml' }, ctx)).rejects.toThrow(
'Failed to parse included file invalid.yaml, YAMLSyntaxError: Flow sequence contains an unexpected }',
);
});
it('should reject invalid secrets', async () => {
await expect(readSecret('hello' as any, ctx)).rejects.toThrow(
'secret must be a `object` type, but the final value was: `"hello"`.',
);
await expect(readSecret({}, ctx)).rejects.toThrow(
"Secret must contain one of 'file', 'env', 'data'",
);
await expect(readSecret({ unknown: 'derp' }, ctx)).rejects.toThrow(
"Secret must contain one of 'file', 'env', 'data'",
);
await expect(readSecret({ data: 'no-data.yml' }, ctx)).rejects.toThrow(
"Invalid format for data secret value, must be of the form <filepath>#<datapath>, got 'no-data.yml'",
);
await expect(
readSecret({ data: 'no-parser.js', path: '.' }, ctx),
).rejects.toThrow('No data secret parser available for extension .js');
await expect(
readSecret({ data: 'my-data.yaml', path: 'some.wrong.yaml.key' }, ctx),
).rejects.toThrow('Value is not an object at some.wrong in my-data.yaml');
});
it('should have 100% test coverage', async () => {
let firstVisit = true;
const secret = {};
const proto = {
get file() {
if (!firstVisit) {
Object.setPrototypeOf(secret, {});
}
firstVisit = false;
return 'a-file';
},
};
Object.setPrototypeOf(secret, proto);
await expect(readSecret(secret, ctx)).rejects.toThrow(
'Secret was left unhandled',
);
});
});
packages/config-loader/src/lib/secrets.ts
-180
View File
@@ -1,180 +0,0 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import * as yup from 'yup';
import yaml from 'yaml';
import { extname } from 'path';
import { JsonObject, JsonValue } from '@backstage/config';
import { isObject, isNever } from './utils';
import { ReaderContext } from './types';
// Reads a file and forwards the contents as is, assuming ut8 encoding
type FileSecret = {
// Path to the secret file, relative to the config file.
file: string;
};
// Reads the secret from an environment variable.
type EnvSecret = {
// The name of the environment file.
env: string;
};
// Reads a secret from a json-like file and extracts a value at a path.
// The supported extensions are define in dataSecretParser below.
type DataSecret = {
// Path to the data secret file, relative to the config file.
data: string;
// The path to the value inside the data file, each element separated by '.'.
path?: string;
};
// TODO(Rugvip): Move this out of secret reading when we remove the deprecated DataSecret and $secret format
type IncludeSecret = {
include: string;
};
type Secret = FileSecret | EnvSecret | DataSecret | IncludeSecret;
// Schema for each type of secret description
const secretLoaderSchemas = {
file: yup.object({
file: yup.string().required(),
}),
env: yup.object({
env: yup.string().required(),
}),
data: yup.object({
data: yup.string().required(),
}),
include: yup.object({
include: yup.string().required(),
}),
};
// The top-level secret schema, which figures out what type of secret it is.
const secretSchema = yup.lazy<object | undefined>(value => {
if (typeof value !== 'object' || value === null) {
return yup.object().required().label('secret');
}
const loaderTypes = Object.keys(
secretLoaderSchemas,
) as (keyof typeof secretLoaderSchemas)[];
for (const key of loaderTypes) {
if (key in value) {
return secretLoaderSchemas[key];
}
}
throw new yup.ValidationError(
`Secret must contain one of '${loaderTypes.join("', '")}'`,
value,
'$secret',
);
});
// Parsers for each type of data secret file.
const dataSecretParser: {
[ext in string]: (content: string) => Promise<JsonObject>;
} = {
'.json': async content => JSON.parse(content),
'.yaml': async content => yaml.parse(content),
'.yml': async content => yaml.parse(content),
};
/**
* Transforms a secret description into the actual secret value.
*/
export async function readSecret(
data: JsonObject,
ctx: ReaderContext,
): Promise<JsonValue | undefined> {
const secret = secretSchema.validateSync(data, { strict: true }) as Secret;
if ('file' in secret) {
return ctx.readFile(secret.file);
}
if ('env' in secret) {
return ctx.env[secret.env];
}
if ('data' in secret) {
console.warn(
`Configuration uses deprecated $data key, use $include instead.`,
);
const url =
'path' in secret ? `${secret.data}#${secret.path}` : secret.data;
const [filePath, dataPath] = url.split(/#(.*)/);
if (!dataPath) {
throw new Error(
`Invalid format for data secret value, must be of the form <filepath>#<datapath>, got '${url}'`,
);
}
const ext = extname(filePath);
const parser = dataSecretParser[ext];
if (!parser) {
throw new Error(`No data secret parser available for extension ${ext}`);
}
const content = await ctx.readFile(filePath);
const parts = dataPath.split('.');
let value: JsonValue | undefined = await parser(content);
for (const [index, part] of parts.entries()) {
if (!isObject(value)) {
const errPath = parts.slice(0, index).join('.');
throw new Error(`Value is not an object at ${errPath} in ${filePath}`);
}
value = value[part];
}
return String(value);
}
if ('include' in secret) {
const [filePath, dataPath] = secret.include.split(/#(.*)/);
const ext = extname(filePath);
const parser = dataSecretParser[ext];
if (!parser) {
throw new Error(`No data secret parser available for extension ${ext}`);
}
const content = await ctx.readFile(filePath);
const parts = dataPath ? dataPath.split('.') : [];
let value: JsonValue | undefined;
try {
value = await parser(content);
} catch (error) {
throw new Error(`Failed to parse included file ${filePath}, ${error}`);
}
for (const [index, part] of parts.entries()) {
if (!isObject(value)) {
const errPath = parts.slice(0, index).join('.');
throw new Error(`Value is not an object at ${errPath} in ${filePath}`);
}
value = value[part];
}
return value;
}
isNever<typeof secret>();
throw new Error('Secret was left unhandled');
}
packages/config-loader/src/lib/transform/apply.test.ts
+84
View File
@@ -0,0 +1,84 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { applyConfigTransforms } from './apply';
describe('applyConfigTransforms', () => {
it('should apply not transforms to input', async () => {
const data = applyConfigTransforms(
'',
{
app: {
title: 'Test',
x: 1,
y: [null, true],
z: null,
},
},
[],
);
await expect(data).resolves.toEqual({
app: {
title: 'Test',
x: 1,
y: [true],
},
});
});
it('should throw if input is not an object', async () => {
const config = applyConfigTransforms('', 'not-config', []);
await expect(config).rejects.toThrow('expected object at config root');
});
it('should apply transforms', async () => {
const config = applyConfigTransforms(
'',
{
app: {
title: 'Test',
x: 1,
y: [null, true],
z: null,
},
},
[
async value => {
if (typeof value === 'number') {
return { applied: true, value: value + 1 };
}
return { applied: false };
},
async value => {
if (typeof value === 'string' && value.length > 1) {
return { applied: true, value: value.split('') };
}
return { applied: false };
},
],
);
await expect(config).resolves.toEqual({
app: {
title: ['T', 'e', 's', 't'],
x: 2,
y: [true],
},
});
});
});
packages/config-loader/src/lib/transform/apply.ts
+90
View File
@@ -0,0 +1,90 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { JsonObject, JsonValue } from '@backstage/config';
import { TransformFunc } from './types';
import { isObject } from './utils';
/**
* Applies a set of transforms to raw configuration data.
*/
export async function applyConfigTransforms(
initialDir: string,
input: JsonValue,
transforms: TransformFunc[],
): Promise<JsonObject> {
async function transform(
inputObj: JsonValue,
path: string,
baseDir: string,
): Promise<JsonValue | undefined> {
let obj = inputObj;
let dir = baseDir;
for (const tf of transforms) {
try {
const result = await tf(inputObj, baseDir);
if (result.applied) {
if (result.value === undefined) {
return undefined;
}
obj = result.value;
dir = result.newBaseDir ?? dir;
break;
}
} catch (error) {
throw new Error(`error at ${path}, ${error.message}`);
}
}
if (typeof obj !== 'object') {
return obj;
} else if (obj === null) {
return undefined;
} else if (Array.isArray(obj)) {
const arr = new Array<JsonValue>();
for (const [index, value] of obj.entries()) {
const out = await transform(value, `${path}[${index}]`, dir);
if (out !== undefined) {
arr.push(out);
}
}
return arr;
}
const out: JsonObject = {};
for (const [key, value] of Object.entries(obj)) {
// undefined covers optional fields
if (value !== undefined) {
const result = await transform(value, `${path}.${key}`, dir);
if (result !== undefined) {
out[key] = result;
}
}
}
return out;
}
const finalData = await transform(input, '', initialDir);
if (!isObject(finalData)) {
throw new TypeError('expected object at config root');
}
return finalData;
}
packages/config-loader/src/lib/transform/include.test.ts
+133
View File
@@ -0,0 +1,133 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { createIncludeTransform } from './include';
const env = jest.fn(async (name: string) => {
return ({
SECRET: 'my-secret',
} as { [name: string]: string })[name];
});
const readFile = jest.fn(async (path: string) => {
const content = ({
'/my-secret': 'secret',
'/my-data.json': '{"a":{"b":{"c":42}}}',
'/my-data.yaml': 'some:\n yaml:\n key: 7',
'/my-data.yml': 'different: { key: hello }',
'/invalid.yaml': 'foo: [}',
} as { [key: string]: string })[path];
if (!content) {
throw new Error('File not found!');
}
return content;
});
const includeTransform = createIncludeTransform(env, readFile);
describe('includeTransform', () => {
it('should not transform unknown values', async () => {
await expect(includeTransform('foo', '/')).resolves.toEqual({
applied: false,
});
await expect(includeTransform([1], '/')).resolves.toEqual({
applied: false,
});
await expect(includeTransform(1, '/')).resolves.toEqual({ applied: false });
await expect(includeTransform({ x: 'y' }, '/')).resolves.toEqual({
applied: false,
});
await expect(includeTransform(null, '/')).resolves.toEqual({
applied: false,
});
});
it('should include text files', async () => {
await expect(
includeTransform({ $file: 'my-secret' }, '/'),
).resolves.toEqual({ applied: true, value: 'secret' });
await expect(includeTransform({ $file: 'no-secret' }, '/')).rejects.toThrow(
'File not found!',
);
});
it('should include env vars', async () => {
await expect(includeTransform({ $env: 'SECRET' }, '/')).resolves.toEqual({
applied: true,
value: 'my-secret',
});
await expect(includeTransform({ $env: 'NO_SECRET' }, '/')).resolves.toEqual(
{
applied: true,
value: undefined,
},
);
});
it('should include config files', async () => {
// New format with path in fragment
await expect(
includeTransform({ $include: 'my-data.json#a.b.c' }, '/'),
).resolves.toEqual({ applied: true, value: 42 });
await expect(
includeTransform({ $include: 'my-data.json#a.b' }, '/'),
).resolves.toEqual({ applied: true, value: { c: 42 } });
await expect(
includeTransform({ $include: 'my-data.yaml#some.yaml.key' }, '/'),
).resolves.toEqual({ applied: true, value: 7 });
await expect(
includeTransform({ $include: 'my-data.yaml' }, '/'),
).resolves.toEqual({
applied: true,
value: {
some: { yaml: { key: 7 } },
},
});
await expect(
includeTransform({ $include: 'my-data.yaml#' }, '/'),
).resolves.toEqual({
applied: true,
value: {
some: { yaml: { key: 7 } },
},
});
await expect(
includeTransform({ $include: 'my-data.yml#different.key' }, '/'),
).resolves.toEqual({ applied: true, value: 'hello' });
});
it('should reject invalid includes', async () => {
await expect(
includeTransform({ $include: 'no-parser.js' }, '/'),
).rejects.toThrow(
'no configuration parser available for included file no-parser.js',
);
await expect(
includeTransform({ $include: 'no-data.yml#different.key' }, '/'),
).rejects.toThrow('File not found!');
await expect(
includeTransform({ $include: 'my-data.yml#missing.key' }, '/'),
).rejects.toThrow(
"value at 'missing' in included file my-data.yml is not an object",
);
await expect(
includeTransform({ $include: 'invalid.yaml' }, '/'),
).rejects.toThrow(
'failed to parse included file invalid.yaml, YAMLSyntaxError: Flow sequence contains an unexpected }',
);
});
});
packages/config-loader/src/lib/transform/include.ts
+124
View File
@@ -0,0 +1,124 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import yaml from 'yaml';
import { extname, dirname, resolve as resolvePath } from 'path';
import { JsonObject, JsonValue } from '@backstage/config';
import { isObject } from './utils';
import { TransformFunc, EnvFunc, ReadFileFunc } from './types';
// Parsers for each type of included file
const includeFileParser: {
[ext in string]: (content: string) => Promise<JsonObject>;
} = {
'.json': async content => JSON.parse(content),
'.yaml': async content => yaml.parse(content),
'.yml': async content => yaml.parse(content),
};
/**
* Transforms a include description into the actual included value.
*/
export function createIncludeTransform(
env: EnvFunc,
readFile: ReadFileFunc,
): TransformFunc {
return async (input: JsonValue, baseDir: string) => {
if (!isObject(input)) {
return { applied: false };
}
// Check if there's any key that starts with a '$', in that case we treat
// this entire object as an include description.
const [includeKey] = Object.keys(input).filter(key => key.startsWith('$'));
if (includeKey) {
if (Object.keys(input).length !== 1) {
throw new Error(
`include key ${includeKey} should not have adjacent keys`,
);
}
} else {
return { applied: false };
}
const includeValue = input[includeKey];
if (typeof includeValue !== 'string') {
throw new Error(`${includeKey} include value is not a string`);
}
switch (includeKey) {
case '$file':
try {
const value = await readFile(resolvePath(baseDir, includeValue));
return { applied: true, value };
} catch (error) {
throw new Error(`failed to read file ${includeValue}, ${error}`);
}
case '$env':
try {
return { applied: true, value: await env(includeValue) };
} catch (error) {
throw new Error(`failed to read env ${includeValue}, ${error}`);
}
case '$include': {
const [filePath, dataPath] = includeValue.split(/#(.*)/);
const ext = extname(filePath);
const parser = includeFileParser[ext];
if (!parser) {
throw new Error(
`no configuration parser available for included file ${filePath}`,
);
}
const path = resolvePath(baseDir, filePath);
const content = await readFile(path);
const newBaseDir = dirname(path);
const parts = dataPath ? dataPath.split('.') : [];
let value: JsonValue | undefined;
try {
value = await parser(content);
} catch (error) {
throw new Error(
`failed to parse included file ${filePath}, ${error}`,
);
}
// This bit handles selecting a subtree in the included file, if a path was provided after a #
for (const [index, part] of parts.entries()) {
if (!isObject(value)) {
const errPath = parts.slice(0, index).join('.');
throw new Error(
`value at '${errPath}' in included file ${filePath} is not an object`,
);
}
value = value[part];
}
return {
applied: true,
value,
newBaseDir: newBaseDir !== baseDir ? newBaseDir : undefined,
};
}
default:
throw new Error(`unknown include ${includeKey}`);
}
};
}
packages/config-loader/src/lib/transform/index.ts
+19
View File
@@ -0,0 +1,19 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
export { applyConfigTransforms } from './apply';
export { createIncludeTransform } from './include';
export { createSubstitutionTransform } from './substitution';
packages/config-loader/src/lib/transform/substitution.test.ts
+66
View File
@@ -0,0 +1,66 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { createSubstitutionTransform } from './substitution';
const env = jest.fn(async (name: string) => {
return ({
SECRET: 'my-secret',
TOKEN: 'my-token',
} as { [name: string]: string })[name];
});
const substituteTransform = createSubstitutionTransform(env);
describe('substituteTransform', () => {
it('should not transform unknown values', async () => {
await expect(substituteTransform(false, '/')).resolves.toEqual({
applied: false,
});
await expect(substituteTransform([1], '/')).resolves.toEqual({
applied: false,
});
await expect(substituteTransform(1, '/')).resolves.toEqual({
applied: false,
});
await expect(substituteTransform({ x: 'y' }, '/')).resolves.toEqual({
applied: false,
});
await expect(substituteTransform(null, '/')).resolves.toEqual({
applied: false,
});
});
it('should substitute env var', async () => {
await expect(substituteTransform('hello ${SECRET}', '/')).resolves.toEqual({
applied: true,
value: 'hello my-secret',
});
await expect(
substituteTransform('${SECRET } $${} ${TOKEN }', '/'),
).resolves.toEqual({ applied: true, value: 'my-secret $${} my-token' });
await expect(substituteTransform('foo ${MISSING}', '/')).resolves.toEqual({
applied: true,
value: undefined,
});
await expect(
substituteTransform('foo ${MISSING} ${SECRET}', '/'),
).resolves.toEqual({ applied: true, value: undefined });
await expect(
substituteTransform('foo ${SECRET} ${SECRET}', '/'),
).resolves.toEqual({ applied: true, value: 'foo my-secret my-secret' });
});
});
packages/config-loader/src/lib/transform/substitution.ts
+40
View File
@@ -0,0 +1,40 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { JsonValue } from '@backstage/config';
import { TransformFunc, EnvFunc } from './types';
/**
* A environment variable substitution transform that transforms e.g. 'token ${MY_TOKEN}'
* to 'token abc' if MY_TOKEN is 'abc'. If any of the substituted variables are undefined,
* the entire expression ends up undefined.
*/
export function createSubstitutionTransform(env: EnvFunc): TransformFunc {
return async (input: JsonValue) => {
if (typeof input !== 'string') {
return { applied: false };
}
const parts: (string | undefined)[] = input.split(/(?<!\$)\$\{([^{}]+)\}/);
for (let i = 1; i < parts.length; i += 2) {
parts[i] = await env(parts[i]!.trim());
}
if (parts.some(part => part === undefined)) {
return { applied: true, value: undefined };
}
return { applied: true, value: parts.join('') };
};
}
packages/config-loader/src/lib/types.ts → packages/config-loader/src/lib/transform/types.ts
+16 -14
View File
@@ -14,20 +14,22 @@
* limitations under the License.
*/
import { JsonObject, JsonValue } from '@backstage/config';
import { JsonValue } from '@backstage/config';
export type EnvFunc = (name: string) => Promise<string | undefined>;
export type ReadFileFunc = (path: string) => Promise<string>;
export type ReadSecretFunc = (
path: string,
desc: JsonObject,
) => Promise<JsonValue | undefined>;
export type SkipFunc = (path: string) => boolean;
/**
* Common context that provides all the necessary hooks for reading configuration files.
*/
export type ReaderContext = {
env: { [name in string]?: string };
readFile: ReadFileFunc;
readSecret: ReadSecretFunc;
};
export type TransformFunc = (
value: JsonValue,
baseDir: string,
) => Promise<
| {
applied: false;
}
| {
applied: true;
value: JsonValue | undefined;
newBaseDir?: string | undefined;
}
>;
packages/config-loader/src/lib/utils.ts → packages/config-loader/src/lib/transform/utils.ts
-5
View File
@@ -24,8 +24,3 @@ export function isObject(obj: JsonValue | undefined): obj is JsonObject {
}
return obj !== null;
}
// A thing to make sure we've narrowed the type down to never
export function isNever<T extends never>() {
return void 0 as T;
}
packages/config-loader/src/loader.test.ts
+22
View File
@@ -19,6 +19,8 @@ import mockFs from 'mock-fs';
describe('loadConfig', () => {
beforeAll(() => {
process.env.MY_SECRET = 'is-secret';
mockFs({
'/root/app-config.yaml': `
app:
@@ -29,8 +31,20 @@ describe('loadConfig', () => {
'/root/app-config.development.yaml': `
app:
sessionKey: development-key
backend:
$include: ./included.yaml
other:
$include: secrets/included.yaml
`,
'/root/secrets/session-key.txt': 'abc123',
'/root/secrets/included.yaml': `
secret:
$file: session-key.txt
`,
'/root/included.yaml': `
foo:
bar: token \${MY_SECRET}
`,
});
});
@@ -104,6 +118,14 @@ describe('loadConfig', () => {
app: {
sessionKey: 'development-key',
},
backend: {
foo: {
bar: 'token is-secret',
},
},
other: {
secret: 'abc123',
},
},
},
]);
packages/config-loader/src/loader.ts
+34 -49
View File
@@ -15,9 +15,16 @@
*/
import fs from 'fs-extra';
import { resolve as resolvePath, dirname, isAbsolute } from 'path';
import { AppConfig, JsonObject, JsonValue } from '@backstage/config';
import { readConfigFile, readEnvConfig, readSecret } from './lib';
import yaml from 'yaml';
import { resolve as resolvePath, dirname, isAbsolute, basename } from 'path';
import { AppConfig } from '@backstage/config';
import {
applyConfigTransforms,
readEnvConfig,
createIncludeTransform,
createSubstitutionTransform,
} from './lib';
import { EnvFunc } from './lib/transform/types';
export type LoadConfigOptions = {
// The root directory of the config loading context. Used to find default configs.
@@ -26,39 +33,22 @@ export type LoadConfigOptions = {
// Absolute paths to load config files from. Configs from earlier paths have lower priority.
configPaths: string[];
// TODO(Rugvip): This will be removed in the future, but for now we use it to warn about possible mistakes.
env: string;
/** @deprecated This option has been removed */
env?: string;
/**
* Custom environment variable loading function
*
* @experimental This API is not stable and may change at any point
*/
experimentalEnvFunc?: EnvFunc;
};
class Context {
constructor(
private readonly options: {
env: { [name in string]?: string };
rootPath: string;
},
) {}
get env() {
return this.options.env;
}
async readFile(path: string): Promise<string> {
return fs.readFile(resolvePath(this.options.rootPath, path), 'utf8');
}
async readSecret(
_path: string,
desc: JsonObject,
): Promise<JsonValue | undefined> {
return readSecret(desc, this);
}
}
export async function loadConfig(
options: LoadConfigOptions,
): Promise<AppConfig[]> {
const configs = [];
const { configRoot } = options;
const { configRoot, experimentalEnvFunc: envFunc } = options;
const configPaths = options.configPaths.slice();
// If no paths are provided, we default to reading
@@ -70,36 +60,31 @@ export async function loadConfig(
if (await fs.pathExists(localConfig)) {
configPaths.push(localConfig);
}
const envFile = `app-config.${options.env}.yaml`;
if (await fs.pathExists(resolvePath(configRoot, envFile))) {
console.error(
`Env config file '${envFile}' is not loaded as APP_ENV and NODE_ENV-based config loading has been removed`,
);
console.error(
`To load the config file, use --config <path>, listing every config file that you want to load`,
);
}
}
const env = envFunc ?? (async (name: string) => process.env[name]);
try {
for (const configPath of configPaths) {
if (!isAbsolute(configPath)) {
throw new Error(`Config load path is not absolute: '${configPath}'`);
}
const config = await readConfigFile(
configPath,
new Context({
env: process.env,
rootPath: dirname(configPath),
}),
);
configs.push(config);
const dir = dirname(configPath);
const readFile = (path: string) =>
fs.readFile(resolvePath(dir, path), 'utf8');
const input = yaml.parse(await readFile(configPath));
const data = await applyConfigTransforms(dir, input, [
createIncludeTransform(env, readFile),
createSubstitutionTransform(env),
]);
configs.push({ data, context: basename(configPath) });
}
} catch (error) {
throw new Error(
`Failed to read static configuration file: ${error.message}`,
`Failed to read static configuration file, ${error.message}`,
);
}