🎯 Top Personalized Recommendations
University of Salento, L
Why we think this paper is great for you:
This paper directly addresses the formalization and enhancement of design patterns, offering insights into evolving principles for complex systems. It provides a systematic framework that you might find highly relevant to your work.
This paper directly addresses the formalization and enhancement of design patterns, offering insights into evolving principles for complex systems. It provides a systematic framework that you might find highly relevant to your work.
Abstract
Traditional ETL and ELT design patterns struggle to meet modern requirements of scalability, governance, and real-time data processing. Hybrid approaches such as ETLT (Extract-Transform-Load-Transform) and ELTL (Extract-Load-Transform-Load) are already used in practice, but the literature lacks best practices and formal recognition of these approaches as design patterns. This paper formalizes ETLT and ELTL as reusable design patterns by codifying implicit best practices and introduces enhanced variants, ETLT++ and ELTL++, to address persistent gaps in governance, quality assurance, and observability. We define ETLT and ELTL patterns systematically within a design pattern framework, outlining their structure, trade-offs, and use cases. Building on this foundation, we extend them into ETLT++ and ELTL++ by embedding explicit contracts, versioning, semantic curation, and continuous monitoring as mandatory design obligations. The proposed framework offers practitioners a structured roadmap to build auditable, scalable, and cost-efficient pipelines, unifying quality enforcement, lineage, and usability across multi-cloud and real-time contexts. By formalizing ETLT and ELTL, and enhancing them through ETLT++ and ELTL++, this work bridges the gap between ad hoc practice and systematic design, providing a reusable foundation for modern, trustworthy data engineering.
Traditional ETL and ELT design patterns struggle to meet modern requirements of scalability, governance, and real-time data processing. Hybrid approaches such as ETLT (Extract-Transform-Load-Transform) and ELTL (Extract-Load-Transform-Load) are already used in practice, but the literature lacks best practices and formal recognition of these approaches as design patterns. This paper formalizes ETLT and ELTL as reusable design patterns by codifying implicit best practices and introduces enhanced variants, ETLT++ and ELTL++, to address persistent gaps in governance, quality assurance, and observability. We define ETLT and ELTL patterns systematically within a design pattern framework, outlining their structure, trade-offs, and use cases. Building on this foundation, we extend them into ETLT++ and ELTL++ by embedding explicit contracts, versioning, semantic curation, and continuous monitoring as mandatory design obligations. The proposed framework offers practitioners a structured roadmap to build auditable, scalable, and cost-efficient pipelines, unifying quality enforcement, lineage, and usability across multi-cloud and real-time contexts. By formalizing ETLT and ELTL, and enhancing them through ETLT++ and ELTL++, this work bridges the gap between ad hoc practice and systematic design, providing a reusable foundation for modern, trustworthy data engineering.
AI Summary
- Formalization of ETLT and ELTL as distinct hybrid design patterns provides a structured approach to reconciling operational trade-offs in modern data environments, such as compute locality, raw data retention, and governance alignment. [2]
- ETLT++ introduces mandatory data contracts at ingress, ensuring early validation and quarantining of non-compliant data based on hard rules, thereby preventing corrupted data propagation into downstream systems. [2]
- ELTL++ mandates managed raw layer storage with metadata-driven ingestion, retention, and tiering policies, effectively combating 'data swamps' and optimizing storage costs by moving older data to colder tiers. [2]
- Both ETLT++ and ELTL++ enforce versioned, append-only loading, guaranteeing full data lineage, auditability, and the ability to reproduce historical analyses by allowing 'time-travel' through dataset states. [2]
- The proposed patterns embed continuous monitoring, semantic curation, and rewindable business logic, transforming ad-hoc data pipelines into reliable, transparent, and auditable systems with measurable service-level objectives. [2]
- The framework addresses the industry-academia gap by providing systematic, reusable design patterns for complex enterprise data challenges, moving beyond isolated theoretical optimizations to practical, operationalizable solutions. [2]
- ETLT (Extract-Transform-Load-Transform): A multi-stage integration paradigm that decouples data quality operations (T1) from business-specific transformations (T2), with an intermediate load stage. [2]
- ELTL (Extract-Load-Transform-Load): A dual-loading architecture that emphasizes preservation of raw data (L1) alongside performance-optimized outputs (L2) after a comprehensive transformation stage. [2]
- ETLT++: An enhanced ETLT pattern incorporating mandatory data contracts, versioned raw storage, rewindable business logic, and continuous monitoring as explicit design obligations. [2]
- ELTL++: An enhanced ELTL pattern featuring smart raw data management (L1), standardized and versioned transformations, dual loading with a curated semantic layer, and embedded governance and observability. [2]
University of Nebraska,Om
Why we think this paper is great for you:
This work delves into the semantics and advanced language constructs of a declarative programming paradigm. It aligns well with your interest in how programming languages are designed and structured.
This work delves into the semantics and advanced language constructs of a declarative programming paradigm. It aligns well with your interest in how programming languages are designed and structured.
Abstract
Modern answer set programming solvers such as CLINGO support advanced language constructs that improve the expressivity and conciseness of logic programs. Conditional literals are one such construct. They form "subformulas" that behave as nested implications within the bodies of logic rules. Their inclusion brings the form of rules closer to the less restrictive syntax of first-order logic. These qualities make conditional literals useful tools for knowledge representation. In this paper, we propose a semantics for logic programs with conditional literals and arithmetic based on the SM operator. These semantics do not require grounding, unlike the established semantics for such programs that relies on a translation to infinitary propositional logic. The main result of this paper establishes the precise correspondence between the proposed and existing semantics.
Modern answer set programming solvers such as CLINGO support advanced language constructs that improve the expressivity and conciseness of logic programs. Conditional literals are one such construct. They form "subformulas" that behave as nested implications within the bodies of logic rules. Their inclusion brings the form of rules closer to the less restrictive syntax of first-order logic. These qualities make conditional literals useful tools for knowledge representation. In this paper, we propose a semantics for logic programs with conditional literals and arithmetic based on the SM operator. These semantics do not require grounding, unlike the established semantics for such programs that relies on a translation to infinitary propositional logic. The main result of this paper establishes the precise correspondence between the proposed and existing semantics.
Institute of Software
Why we think this paper is great for you:
This paper explores a novel approach to compiler design using large language models, directly connecting with the implementation and evolution of programming languages. You will find its discussion on transforming source code particularly interesting.
This paper explores a novel approach to compiler design using large language models, directly connecting with the implementation and evolution of programming languages. You will find its discussion on transforming source code particularly interesting.
Abstract
In recent years, end-to-end Large Language Model (LLM) technology has shown substantial advantages across various domains. As critical system software and infrastructure, compilers are responsible for transforming source code into target code. While LLMs have been leveraged to assist in compiler development and maintenance, their potential as an end-to-end compiler remains largely unexplored. This paper explores the feasibility of LLM as a Compiler (LaaC) and its future directions. We designed the CompilerEval dataset and framework specifically to evaluate the capabilities of mainstream LLMs in source code comprehension and assembly code generation. In the evaluation, we analyzed various errors, explored multiple methods to improve LLM-generated code, and evaluated cross-platform compilation capabilities. Experimental results demonstrate that LLMs exhibit basic capabilities as compilers but currently achieve low compilation success rates. By optimizing prompts, scaling up the model, and incorporating reasoning methods, the quality of assembly code generated by LLMs can be significantly enhanced. Based on these findings, we maintain an optimistic outlook for LaaC and propose practical architectural designs and future research directions. We believe that with targeted training, knowledge-rich prompts, and specialized infrastructure, LaaC has the potential to generate high-quality assembly code and drive a paradigm shift in the field of compilation.
In recent years, end-to-end Large Language Model (LLM) technology has shown substantial advantages across various domains. As critical system software and infrastructure, compilers are responsible for transforming source code into target code. While LLMs have been leveraged to assist in compiler development and maintenance, their potential as an end-to-end compiler remains largely unexplored. This paper explores the feasibility of LLM as a Compiler (LaaC) and its future directions. We designed the CompilerEval dataset and framework specifically to evaluate the capabilities of mainstream LLMs in source code comprehension and assembly code generation. In the evaluation, we analyzed various errors, explored multiple methods to improve LLM-generated code, and evaluated cross-platform compilation capabilities. Experimental results demonstrate that LLMs exhibit basic capabilities as compilers but currently achieve low compilation success rates. By optimizing prompts, scaling up the model, and incorporating reasoning methods, the quality of assembly code generated by LLMs can be significantly enhanced. Based on these findings, we maintain an optimistic outlook for LaaC and propose practical architectural designs and future research directions. We believe that with targeted training, knowledge-rich prompts, and specialized infrastructure, LaaC has the potential to generate high-quality assembly code and drive a paradigm shift in the field of compilation.
TU Wien
Why we think this paper is great for you:
This paper discusses the role of software models and code generation in modern software engineering, touching upon principles relevant to robust software design and development. It offers a perspective on maintaining quality in AI-driven creation.
This paper discusses the role of software models and code generation in modern software engineering, touching upon principles relevant to robust software design and development. It offers a perspective on maintaining quality in AI-driven creation.
Abstract
Generative AI enables rapid ``vibe coding," where natural language prompts yield working software systems. While this lowers barriers to software creation, it also collapses the boundary between prototypes and engineered software, leading to fragile systems that lack robustness, security, and maintainability. We argue that this shift motivates a reimagining of software models. Rather than serving only as upfront blueprints, models can be recovered post-hoc from AI-generated code to restore comprehension, expose risks, and guide refinement. In this role, models serve as mediators between human intent, AI generation, and long-term system evolution, providing a path toward sustainable AI-driven software engineering.
Generative AI enables rapid ``vibe coding," where natural language prompts yield working software systems. While this lowers barriers to software creation, it also collapses the boundary between prototypes and engineered software, leading to fragile systems that lack robustness, security, and maintainability. We argue that this shift motivates a reimagining of software models. Rather than serving only as upfront blueprints, models can be recovered post-hoc from AI-generated code to restore comprehension, expose risks, and guide refinement. In this role, models serve as mediators between human intent, AI generation, and long-term system evolution, providing a path toward sustainable AI-driven software engineering.
University of California
Why we think this paper is great for you:
This paper explores the concept of function decomposability in scientific design, which might resonate with your interest in functional concepts and general design principles. It offers insights into structuring complex systems.
This paper explores the concept of function decomposability in scientific design, which might resonate with your interest in functional concepts and general design principles. It offers insights into structuring complex systems.
Abstract
In the era of AI-driven science and engineering, we often want to design discrete objects in silico according to user-specified properties. For example, we may wish to design a protein to bind its target, arrange components within a circuit to minimize latency, or find materials with certain properties. Given a property predictive model, in silico design typically involves training a generative model over the design space (e.g., protein sequence space) to concentrate on designs with the desired properties. Distributional optimization -- which can be formalized as an estimation of distribution algorithm or as reinforcement learning policy optimization -- finds the generative model that maximizes an objective function in expectation. Optimizing a distribution over discrete-valued designs is in general challenging because of the combinatorial nature of the design space. However, many property predictors in scientific applications are decomposable in the sense that they can be factorized over design variables in a way that could in principle enable more effective optimization. For example, amino acids at a catalytic site of a protein may only loosely interact with amino acids of the rest of the protein to achieve maximal catalytic activity. Current distributional optimization algorithms are unable to make use of such decomposability structure. Herein, we propose and demonstrate use of a new distributional optimization algorithm, Decomposition-Aware Distributional Optimization (DADO), that can leverage any decomposability defined by a junction tree on the design variables, to make optimization more efficient. At its core, DADO employs a soft-factorized "search distribution" -- a learned generative model -- for efficient navigation of the search space, invoking graph message-passing to coordinate optimization across linked factors.
In the era of AI-driven science and engineering, we often want to design discrete objects in silico according to user-specified properties. For example, we may wish to design a protein to bind its target, arrange components within a circuit to minimize latency, or find materials with certain properties. Given a property predictive model, in silico design typically involves training a generative model over the design space (e.g., protein sequence space) to concentrate on designs with the desired properties. Distributional optimization -- which can be formalized as an estimation of distribution algorithm or as reinforcement learning policy optimization -- finds the generative model that maximizes an objective function in expectation. Optimizing a distribution over discrete-valued designs is in general challenging because of the combinatorial nature of the design space. However, many property predictors in scientific applications are decomposable in the sense that they can be factorized over design variables in a way that could in principle enable more effective optimization. For example, amino acids at a catalytic site of a protein may only loosely interact with amino acids of the rest of the protein to achieve maximal catalytic activity. Current distributional optimization algorithms are unable to make use of such decomposability structure. Herein, we propose and demonstrate use of a new distributional optimization algorithm, Decomposition-Aware Distributional Optimization (DADO), that can leverage any decomposability defined by a junction tree on the design variables, to make optimization more efficient. At its core, DADO employs a soft-factorized "search distribution" -- a learned generative model -- for efficient navigation of the search space, invoking graph message-passing to coordinate optimization across linked factors.
Yale University, Brown Un
Why we think this paper is great for you:
While from a different domain, this paper's focus on 'designs' and systematic approaches could offer a broader perspective on structured problem-solving. It details a step-by-step guide to specific design patterns within its field.
While from a different domain, this paper's focus on 'designs' and systematic approaches could offer a broader perspective on structured problem-solving. It details a step-by-step guide to specific design patterns within its field.
Abstract
We develop a step-by-step guide to leniency (a.k.a. judge or examiner instrument) designs, drawing on recent econometric literatures. The unbiased jackknife instrumental variables estimator (UJIVE) is purpose-built for leveraging exogenous leniency variation, avoiding subtle biases even in the presence of many decision-makers or controls. We show how UJIVE can also be used to assess key assumptions underlying leniency designs, including quasi-random assignment and average first-stage monotonicity, and to probe the external validity of treatment effect estimates. We further discuss statistical inference, arguing that non-clustered standard errors are often appropriate. A reanalysis of Farre-Mensa et al. (2020), using quasi-random examiner assignment to estimate the value of patents to startups, illustrates our checklist.
We develop a step-by-step guide to leniency (a.k.a. judge or examiner instrument) designs, drawing on recent econometric literatures. The unbiased jackknife instrumental variables estimator (UJIVE) is purpose-built for leveraging exogenous leniency variation, avoiding subtle biases even in the presence of many decision-makers or controls. We show how UJIVE can also be used to assess key assumptions underlying leniency designs, including quasi-random assignment and average first-stage monotonicity, and to probe the external validity of treatment effect estimates. We further discuss statistical inference, arguing that non-clustered standard errors are often appropriate. A reanalysis of Farre-Mensa et al. (2020), using quasi-random examiner assignment to estimate the value of patents to startups, illustrates our checklist.
Fakultt fr Mathematik
Why we think this paper is great for you:
This paper investigates the feasibility of certain 'programs,' which, in a very abstract sense, involves defining and solving structured problems. It explores the underlying logic of system constraints and optimal solutions.
This paper investigates the feasibility of certain 'programs,' which, in a very abstract sense, involves defining and solving structured problems. It explores the underlying logic of system constraints and optimal solutions.
Abstract
We investigate the feasibility problem for generalized inverse linear programs. Given an LP with affinely parametrized objective function and right-hand side as well as a target set Y, the goal is to decide whether the parameters can be chosen such that there exists an optimal solution that belongs to Y (optimistic scenario) or such that all optimal solutions belong to Y (pessimistic scenario). We study the complexity of this decision problem and show how it depends on the structure of the set Y, the form of the LP, the adjustable parameters, and the underlying scenario. For a target singleton Y = {y}, we show that the problem is tractable if the given LP is in standard form, but NP-hard if the LP is given in natural form. If instead we are given a target basis B, the problem in standard form becomes NP-complete in the optimistic case, while remaining tractable in the pessimistic case. For partially fixed target solutions, the problem gets almost immediately NP-hard, but we prove fixed-parameter tractability in the number of non-fixed variables. Moreover, we give a rigorous proof of membership in NP for any polyhedral target set, and discuss how this property can be extended to more general target sets using an oracle-based approach.
We investigate the feasibility problem for generalized inverse linear programs. Given an LP with affinely parametrized objective function and right-hand side as well as a target set Y, the goal is to decide whether the parameters can be chosen such that there exists an optimal solution that belongs to Y (optimistic scenario) or such that all optimal solutions belong to Y (pessimistic scenario). We study the complexity of this decision problem and show how it depends on the structure of the set Y, the form of the LP, the adjustable parameters, and the underlying scenario. For a target singleton Y = {y}, we show that the problem is tractable if the given LP is in standard form, but NP-hard if the LP is given in natural form. If instead we are given a target basis B, the problem in standard form becomes NP-complete in the optimistic case, while remaining tractable in the pessimistic case. For partially fixed target solutions, the problem gets almost immediately NP-hard, but we prove fixed-parameter tractability in the number of non-fixed variables. Moreover, we give a rigorous proof of membership in NP for any polyhedral target set, and discuss how this property can be extended to more general target sets using an oracle-based approach.
Interests not found
We did not find any papers that match the below interests.
Try other terms also consider if the content exists in arxiv.org.
- Object Oriented Programming
You can edit or add more interests any time.
Help us improve your experience!
This project is on its early stages your feedback can be pivotal on the future of the project.
Let us know what you think about this week's papers and suggestions!
Give Feedback