WASHINGTON (CNN) -

President Barack Obama talked about his own family's struggle to balance work and family life at Monday's White House Summit on Working Families, calling on Congress to act to make finding that balance easier.

"Before I moved into the White House, I was away a lot, sometimes with work, sometimes with campaigning. Michelle was working full-time and was at home with the responsibility all too often of dealing with everything that the girls needed," the President said.

"If Malia or Sasha got sick, or the babysitter did not show up, it was Michelle who got the call. And, fortunately, she had an employer who understood if she needed to leave work in the middle of the day or change her schedule suddenly."

The President said that expanding flexibility in the workplace and having paid parental leave would help America compete on the world stage, noting that the United States is the only developed country to not offer paid maternity leave.

"When a new baby arrives or an aging parent gets sick, workers have to make painful decisions about whether they can afford to be there when their families need them the most. Many women can't even get a paid day off to give birth. Now, that's a pretty low bar," Obama said.

He recalled being a father of a newborn with late-night feedings and the importance of being able to be with his daughters immediately after their births.

"I was lucky enough to be able to take some time off so that I was there for the 2 a.m. feeding and the soothing, and just getting to know them and making sure they knew me. And that bond is irreplaceable. And I want every father and every child to have that opportunity," he said.

During his remarks, the President announced he would sign a presidential memorandum for federal agencies to expand access to work schedule flexibility for all federal employees. The White House later Monday informed government agencies that the President had done so.

In a memo obtained by CNN from a White House official, Chief of Staff Denis McDonough said as the President called for business to adopt workplace flexibilities, "so, too, must we lead by example."

"(Obama) signed a memorandum directing government agencies to expand the availability of flexible work arrangements, to increase training on their beneficial use, to lift up our successes, and to hold ourselves accountable for our results," McDonough said in informing White House employees of the order.

"The memorandum also calls on agencies to make clear that, no matter our role, we all have the right to request flexible work schedules through thoughtful conversations with our managers. These steps aren't a silver bullet, but they are designed to move us in the right direction. You'll be hearing more from us on these efforts in the coming months."

Obama also urged the passage of the Pregnant Workers Fairness Act, which would require employers to make accommodations for pregnant workers to be able to keep working while pregnant and prevent them from having to take unpaid leave.

He also used the speech to encourage companies to give better access to high-quality childcare and he repeated his call to raise the minimum wage to $10.10 an hour.

He argued that these are not solely issues to be directed at women because they impact the entire family.

"At a time when women are nearly half of our workforce, among our most skilled workers, are the primary breadwinners in more families than ever before, anything that makes life harder for women makes life harder for families and makes life harder for children," Obama said. "When women succeed, America succeeds, so there's no such thing as a women's issue."

The summit, held Monday at a Washington hotel, featured first lady Michelle Obama, Vice President Joe Biden and his wife, Jill Biden, senior adviser Valerie Jarrett, Secretary of Labor Thomas Perez, as well as business leaders and educators.

Prior to his speech, Obama made an unannounced stop for lunch at a local Chipotle with four participants at the summit who shared with him their struggles on balancing careers and families.